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Re: [EXTERNAL] Re: COVID-19 Tech Response | Call 
tomorrow? 


From Monika Bickert Б 


To: 'Courtnev Gregoire (CELA)" , Leslie Miller 
і SETT Lauren Culbertson > 


"Parker, Lynne E. EOP/OSTP' >, Tyler Otto 
, Kratsios, Michael J. EOP/OSTP" 


"Fred Humphries (CELA)" 


Cc: Payton Iheme 
Johanna Shelton , Lee Carosi Dunn 
Jessica Ashooh Mark Isakowitz 


Date: Wed, 11 Mar 2020 21:47:19 -0400 


I'll be on for Facebook. Thanks. 


From: Courtney Gregoire (CELA) 
Sent: Wednesday, March 11, 2020 5:56:41 PM 
To: Leslie Miller Lauren Culbertson 


Cc: Parker, Lynne E. EOP/OSTP (БЕЗЕ Tyler Otto (БУ(6 w; 
Kratsios, Michael J. EOP/OSTP(BI(U II Monika Bickert GCN >; 
CXC ANN Frea Humphries (CELA) ШЕШШ ОО; 
Portia Wu (B) Payton heme SGI Johanna Shelton 
DCN Lee Carosi Dunn ӘӘ; Jessica Ashooh 
С GI 


Subject: RE: [EXTERNAL] Re: COVID-19 Tech Response | Call tomorrow? 


Microsoft will join as well. Thanks. 


From: Leslie Miller 
Sent: Wednesday, March 11, 2020 5:53 PM 
To: Lauren Culbertson 


(6) > 
Cc: Parker, Lynne E. EOP/OSTP BST Tyler Otto 
Kratsios, Michael J. EOP/OSTP BWWVWCƏPEPEPEEQécƏBBHBIIGqNGIƏGOII—€IUOIIII€IIIICGCOOurtney Gregoire (CELA) 
rn rais (red Humphries (CELA 
С т=ш= оо O (ET 
Johanna Shelton ӘС Lee Carosi Dunn (BGI Jessica Ashooh 
O Merk so owi — 


Subject: Re: [EXTERNAL] Re: COVID-19 Tech Response | Call tomorrow? 


I'm happy to join as well for YouTube/Google. 


On Wed, Mar 11, 2020 at 5:08 PM Lauren Culbertson (DO wrote: 


Twitter will be on, too. Thanks! 
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On Wed, Mar 11, 2020 at 6:57 PM Parker, Lynne E. EOP/OSTP OTT wrote: 


Very good, thanks. Here are the call-in details: 


Participant Dial-In: SG 
Participant Code: (DO 


Lynne 


Lynne E. Parker, Ph.D. 
Deputy Chief Technology Officer of the United States 
The White House 


Office: (SK 


From: Tyler Otto 
Sent: Wednesday, March 11, 2020 6:52 PM 


To: Kratsios, Michael J. EOP/OSTP (б 
; Fred 


L ОЕ ЕЕЕ ЕШ ЕЕЕ ЕЕЕ ЕЕН ЕЕЕ ЕЕ; 
Humphries (CELA) (SG) шо = 
Iheme (GI, Johanna Shelton SIE Lee Carosi Dunn 
JIGU: Lauren Culbertson GG; Parker, Lynne E. 
EOP/OSTP (Бб) И ИШЕ; Jessica Ashooh dB 


Subject: [EXTERNAL] Re: COVID-19 Tech Response | Call tomorrow? 


Reddit can attend the call tomorrow at 12 eastern time. Please send along the call details when 
available. 


Thanks 


Tyler 


On Mar 11, 2020, at 3:16 PM, Kratsios, Michael J. EOP/OSTP 


eu wrote: 


Hi Courtney, Monika, Nick, Leslie, and Tyler, 
At today’s White House COVID-19 Tech Meeting (see attached for the readout), one 


of the important work streams we identified is for better coordination to address 
COVID-19 misinformation on online platforms. 
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One industry collaboration model that might be useful is GIFCT, which you are all 
familiar with. l think you all could provide great insight into workable cross-industry 
mechanisms to facilitate information sharing, and to help us set up something similar 
for COVID-19. And do so very quickly. 


Are you available for a short call tomorrow, say noon eastern? We'd greatly 
appreciate leveraging your expertise on this challenge. 


Many thanks, 
Michael 


Michael Kratsios 

Deputy Assistant to the President 

Chief Technology Officer of the United States 
The White House 

o 

«Mail Attachment.eml» 


Lauren M. Culbertson 
Twitter Public Policy 


Follow me at @lculbs 


Г == EB 


Leslie Miller 
Google LLC 
1600 Amphitheatre Parkway 
Mountain View, CA 94043 
o 
c 
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RE: [EXTERNAL] Re: COVID-19 Tech Response | Call 
tomorrow? 


From “Courtney Gregoire (CELA) (Be, 
To: Leslie Miler BKEJT, Lauren Culbertson (BNEW 


"Parker, Lynne E. EOP/OSTP" , Tyler Otto 
"Kratsios, Michael J. EOP/OSTP" 


, "Fred 


Cc: Humphries (CELA)" 
Payton Iheme Johanna Shelton 
, Jessica Ashooh 


>, Mark Isakowitz 


Date: Wed, 11 Mar 2020 20:56:41 -0400 


Microsoft will join as well. Thanks. 


From: Leslie Miller 
Sent: Wednesday, March 11, 2020 5:53 PM 


To: Lauren Culbertson ЕЗ > 
Cc: Parker, Lynne E. EOP/OSTP (БІВ: Tyler Otto (БУБУ ЕЕЕ: 


Kratsios, Michael J. EOP/OSTP (B(GT Courtney Gregoire (CELA) 

; Fred Humphries (CELA) 

; Portia Wu BDIET Payton Iheme <payton@fb.com>; 
Lee Carosi Dunn BDIET Jessica Ashooh 


Mark Isakowitz > 
Subject: Re: [EXTERNAL] Re: COVID-19 Tech Response | Call tomorrow? 


Johanna Shelton 


I'm happy to join as well for YouTube/Google. 


On Wed, Mar 11, 2020 at 5:08 PM Lauren Сибегіѕог (б ШШШ: wrote: 


Twitter will be on, too. Thanks! 


On Wed, Mar 11, 2020 at 6:57 PM Parker, Lynne Е. ЕОР/ОЗТР (БУ) ШШШ wrote: 


Very good, thanks. Here are the call-in details: 


Participant Dial-In: BG 
Participant Code: (b(6) ` 


Lynne 
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Lvnne E. Parker, Ph.D. 
Deputy Chief Technology Officer of the United States 
The White House 


Office: ББ ШЕЕ 


From: Tyler Otto 
Sent: Wednesday, March 11, 2020 6:52 PM 

To: Kratsios, Michael J. EOP/OSTPIBIUGU > 
oc e 
тхл а. Portia W БО. Payton 
Iheme 4B) ohanna Shelton BIS Lee Carosi Dunn 


; Lauren Culbertson ; Parker, Lynne E. 


EOP/OST Jessica Ashooh (BS 


Subject: [EXTERNAL] Re: COVID-19 Tech Response | Call tomorrow? 


Reddit can attend the call tomorrow at 12 eastern time. Please send along the call details when 
available. 


Thanks 


Tyler 


On Mar 11, 2020, at 3:16 PM, Kratsios, Michael J. EOP/OSTP 


GG: wrote: 


Hi Courtney, Monika, Nick, Leslie, and Tyler, 


At today's White House COVID-19 Tech Meeting (see attached for the readout), one 
of the important work streams we identified is for better coordination to address 
COVID-19 misinformation on online platforms. 


One industry collaboration model that might be useful is GIFCT, which you are all 
familiar with. l think you all could provide great insight into workable cross-industry 
mechanisms to facilitate information sharing, and to help us set up something similar 
for COVID-19. And do so very quickly. 


Are you available for a short call tomorrow, say noon eastern? We'd greatly 
appreciate leveraging your expertise on this challenge. 


Many thanks, 
Michael 


Michael Kratsios 
Deputy Assistant to the President 
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Chief Technology Officer of the United States 
The White House 


<Mail Attachment.eml> 


Lauren M. Culbertson 
Twitter Public Policy 


Follow me at @lculbs 


Г == 


Leslie Miller 

Google LLC 

1600 Amphitheatre Parkway 
Mountain View, CA 94043 
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RE: Google, Amazon, other tech companies meet with White 
House on helping to track coronavirus spread - The 


Washington Post 


То: Nick Sinai BDOTI 
Cc: ||| "Parker, Lynne E. EOPrOSTP'iBEREEEIIIII££III  GIƏOIə 


Date: Wed, 11 Mar 2020 18:28:06 -0400 


Thank you! Feel free to connect us. 


From: Nick Sinai 
Sent: Wednesday, March 11, 2020 1:03 PM 


To: Kratsios, Michael J. EOP/OSTP (БЕ 


Subject: [EXTERNAL] FW: Google, Amazon, other tech companies meet with White House on helping to 
track coronavirus spread - The Washington Post 


Michael, 


I have a company (Vext) that could really help -- especially around information dissemination, 
Q&As, getting info in the large platforms, etc. 


They are a public company -- happy to connect you with the CEO/founder. He wants to 
help/volunteer, but I know how hard it can be to process/evaluate these offers during these times. 


Nick 


From: Nick Sinai 
Sent: Wednesday, March 11, 2020 12:44 PM 
To: Nick Sinai 
Subject: Google, Amazon, other tech companies meet with White House on helping to track 
coronavirus spread - The Washington Post 


>https://www.washingtonpost.com/technology/2020/03/11/white-house-tech-meeting- 
coronavirus/< 


1 
epic.org EPIC-20-03-24-OSTP-FOIA-202007 15-Production-pt3 000218 


The information contained in this message may be privileged and confidential and protected from 
disclosure. If the reader of this message is not the intended recipient, you are hereby notified that 
any dissemination, distribution or copying of this communication is strictly prohibited. If you have 
received this communication in error, please notify us immediately by replying to this message 
and deleting it from your computer. 


Thank you. 


Insight Venture Management, LLC 
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FW: connect -- COVID19 


From "Kratsios, Michael J. EOP/OSTP" 


To: "Parker, Lynne E. EOP/OSTP" DO 


Date: Wed, 11 Mar 2020 14:59:52 -0400 


From: Courtney Gregoire (CELA) (б "I 


Sent: Wednesday, March 11, 2020 2:55 PM 


To: Kratsios, Michael J. EOP/OSTP «ФӘ > 


Subject: [EXTERNAL] RE: connect -- COVID19 
Michael, 


Quick follow-up from our call just now, as | look to convene the folks working on this issue across 
Microsoft. | know that the DC points of contact for the various tech companies would be the right way 
to convene a working group. Although a very different content area, the current Global Internet Forum 
to Counter Terrorism Board representatives are: 

- Myself, current Chair 


- Monika Bickert (B6) Facebook 
- Nick Pickles, БУ 
- — Leslie Miler (BNO 


| mentioned Reddit would be an important company to engage; and | understand a good point of 


contact would be Tyler Otto, i ill 


Thanks, 
Courtney 


Courtney Gregoire 
Chief Digital Safety Officer 
Microsoft Corporation 
Office: 
Mobile: 


From: Kratsios, Michael J. EOP/OSTP 4860] 


Sent: Wednesday, March 11, 2020 8:55 AM 
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To: Courtney Gregoire (CELA) (BGs 


Subject: [EXTERNAL] connect -- COVID19 
Hi Courtney -- Great seeing you last week at the WH. Do you have a second for a quick call today? 


Michael Kratsios 

Deputy Assistant to the President 

Chief Technology Officer of the United States 
The White House 
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RE: [EXTERNAL] Re: COVID-19 Al Dataset 


From: "Parker, Lynne E. EOP/OSTP" (Of > 
To: Anthony Goldbloom (БЕ) Й, 


'Kratsios, Michael J. EOP/OSTP' , Ben Hamner 
, Devvret Rish Sebastian 


Сс: Kohlmeier , "Parker, Lynne E. EOP/OSTP" 
Date: Thu, 12 Mar 2020 18:46:37 -0400 
ОРТ AI-COVID-19-Questions.docx (25.19 kB) 


Lvnne 
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DRAFT || NOT FOR DISTRIBUTION 
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From: 'Hemandez, Elena R. €OP/0¢ TP: Пн 

To: "Baum, Kristina В. EoP/osTP" (Oi 

- "Kratsios, Michael J. EOP/OSTP' , "Parker, Lynne 
: E. EOP/OSTP" > 


Date: Thu, 12 Mar 2020 12:33:12 -0400 


ĠIET Coronavirus Meeting.docx (18.76 kB) 


Attachment 
5: 


From: Baum, Kristina В. ЕОР/О5ТЕ ЕЕЕ 


Sent: Thursday, March 12, 2020 10:12 AM 
To: Hernandez, Elena R. EOP/OSTP 
Subject: 


Hey! 


Best, 
Kristina 


Kristina Baum 

Communications Director 

Office of Science and Technology Policy 
Executive Office of the President 

The White House 

С 

Twitter: @KristinaBaum45 


** Al] emails and replies are subject to the Freedom of Information Act** 
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RE: COVID-19 Tech Response | Call tomorrow? 


From «Parker, Lynne E. EOP/OSTP" Se 


"Kratsios, Michael J. = RE 
То: 


"Fred Humphries (CELA)" , Portia Wu 
Johanna Shelton 
Lauren Culbertson 


, Payton Iheme 
, Lee Carosi Dunn 
"Parker, Lynne E. EOP/OSTP" 


Cc: 


Date: Thu, 12 Mar 2020 08:40:14 -0400 


Confirming today's call at Noon Eastern. 


Participant Dial-In (B6) 
Participant Code: (BG) ННЯ 


Many thanks, all. 


Lynne E. Parker, Ph.D. 
Deputy Chief Technology Officer of the United States 
The White House 


Office: (SCN 
From: Kratsios, Michael J. EOP/OSTP (BU) 


Sent: Wednesday, March 11, 2020 6:16 PM 

LS У (НИ 

Cc: Fred Humphries (CELA) (TT: Portia wu GĦENI: Payton 
; Johanna Shelton >; Lee Carosi Dunn 

Lauren Culbertson >; Parker, Lynne E. EOP/OSTP 


Subject: COVID-19 Tech Response | Call tomorrow? 


Hi Courtney, Monika, Nick, Leslie, and Tyler, 


At today's White House COVID-19 Tech Meeting (see attached for the readout), one of the important 
work streams we identified is for better coordination to address COVID-19 misinformation on online 
platforms. 


One industry collaboration model that might be useful is GIFCT, which you are all familiar with. | think 


youall could provide great insight into workable cross-industry mechanisms to facilitate information 
sharing, and to help us set up something similar for COVID-19. And do so very quickly. 
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Are you available for a short call tomorrow, say noon eastern? We'd greatly appreciate leveraging your 
expertise on this challenge. 


Many thanks, 
Michael 


Michael Kratsios 

Deputy Assistant to the President 

Chief Technology Officer of the United States 
The White House 
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[EXTERNAL] Google comments on AI Regulatory Principles 


From: Kent Wale, 
To: 'Kratsios, Michael J. EOP/OSTP" (BUG) 


Date: Fri, 13 Mar 2020 12:14:26 -0400 
Attachments: Google Submission - Al Regulatory Principles.pdf (171.05 kB) 
Michael -- 


It was good to see you at the Five Eyes event last week. We continue to work to promote a coalition of 
tech companies taking action against CSAM, and appreciate your support for that effort. 


I also wanted to personally congratulate you, OSTP and OMB, and the Administration for putting 

forward a smart, balanced, and coherent national Al policy. Google has submitted favorable comments 
(attached, which will also be posted here), including some constructive suggestions for next steps. We 
believe this framework will help government advance the public interest while encouraging innovation, 
creativity, and economic growth. We've expressed the same thoughts to your colleagues at OIRA. And 
Google and our GAPP team are prepared to support your efforts in this space going forward, sharing our 
technical expertise, policy ideas, and experiences implementing our own AI Principles. 


On a separate note, we have ramped up our response to COVID-19, and look forward to continuing work 
with you on that front. 


Best, 
-- Kent 


Kent Walker 
Senior Vice President, Global Affairs / Google LLC 
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Google 


March 12, 2020 


Mr. Russell Vought 

Acting Director 

Office of Management and Budget 
Washington, DC 20503 


Dear Mr. Vought, 


Google welcomes the opportunity to provide comments in response to the Office of 
Management and Budget's Request for Comments on a Draft Memorandum to the Heads of 
Executive Departments and Agencies, “Guidance for Regulation of Artificial Intelligence 
Applications.” In particular, we support the memorandum's message that artificial 
intelligence (Al) and machine learning (ML) hold extraordinary potential to improve our lives 
and that the responsible development of Al technology should be encouraged, not 
discouraged. Moreover, we agree that federal agencies must avoid creating unnecessary 
barriers to Al development and use by taking a thoughtful, context-specific approach when 
regulating Al that carefully assesses risks and weighs costs and benefits. 


The Promise of Al 


Google's mission is to organize the world's information and make it universally accessible and 
useful. In pursuing this mission, Google is prioritizing investment in advanced technologies 
such as AI/ML. These technologies make Google's core products and services much more 
useful to the public, including Android, Assistant, Cloud, Gmail, Maps, Photos, Pixel, Search, 
YouTube, and many more. 


Google is also creating tools to ensure that everyone can access Al, including researchers and 
developers, entrepreneurs and businesses of all sizes, academics, nonprofits, and 
governments. Wider accessibility is how Al will have its biggest impact and how society can 
reap its full promise. Critical to this approach is open-sourcing Al tools through systems such 
as TensorFlow, a framework which makes machine learning faster, smarter, more flexible, and 
available to a wider community of developers. Google Cloud brings this technology to the 
enterprise world, offering a range of Al-powered products and solutions, from pre-built APIs 
for computer vision and naturallanguage processing to end-to-end solutions that are helping 
to transform sectors such as financial services, retail, healthcare, and beyond. 


Al is making it easier for people to go about their daily lives, from managing household and 
workplace tasks to breaking down language barriers, while also transforming organizations 
across both the private and public sectors. But the future potential is far greater. Al provides 
new hope for addressing the world's toughest problems, from rethinking transportation to 


1 Federal Register Doc. 2020-00261, 
https://www.whitehouse.gov/wp-content/uploads/2020/01/Draft-OMB-Memo-on-Regulation-of-Al-1-7-19.p 
df. 

2 https://www.tensorflow.org/. 
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advancing scientific discovery to tackling complex environmental challenges. That's why 
we're conducting research that advances the state-of-the-art in the field, applying Al across 
many domains. For instance, we've recently pioneered an Al model that can help doctors spot 
breast cancer in mammograms with greater accuracy, and another that makes immediate, 
hyperlocal rainfall forecasts more accurately than existing methods to help everything from 
agricultural productivity to resilience against flooding. 


Using Al Responsibly 


We believe that Al willbe overwhelmingly beneficial for society. But it is critical that the 
technology is developed and used to help people — that it is socially beneficial, fair, 
accountable, and works for everyone. Al must be developed and used responsibly in ways 
that build trust. And potential harms must be acknowledged and mitigated, or prevented. 


As a leader in the development and use of Al, Google has recognized both the enormous 
benefits and the issues that Al raises, and has wrestled with those issues in the context of our 
own operations. To that end, in 2018 we established principles governing our development 
and use of Google А! applications, best practices to share in our work with communities 
outside of Google, and programs to operationalize our efforts. 


Our Al Principles? guide the ethical development and use of Al in our research, products, and 
services. These guidelines help us avoid unfair bias, rigorously review for safety, design with 
privacy top-of-mind, and make the technology accountable to people. They also specify 
areas where we will not design or deploy Al, such as where human rights might be negatively 
impacted. But principles that simply remain on paper are meaningless. So we've developed 
business processes to put them into action, such as requiring vigorous testing of Google's AI 
decisions for fairness and conducting independent assessments of new products against our 
principles. 


We have gone even further and are leading efforts to push for the responsible and socially 
beneficial use of Al applications developed outside of Google. We believe that all 
organizations creating Al tools should adopt guiding principles and robust internal review 
processes. That is why we regularly share our recommended responsible Al practices^ with 
the broader Al community, other companies, nonprofit organizations, and academia. Google 
Cloud has also created commercial tools that enable enterprise organizations to incorporate 
these practices into their own Al applications. For example, we have introduced tools like 
Explainable Al? to help our customers better test and understand the outputs of their models. 
Additionally, we have invested in scaling frameworks like Model Cards' to increase 
transparency and comprehension around the proper use and limitations of Al models. 


What's more, we've worked closely with governments and civil society stakeholders around 
the world to seize opportunities and address challenges related to Al. This includes 
stakeholders at all levels of government here in the United States. Google has provided 


3 https://ai.google/principles/. 

^ https://ai.google/responsibilities/responsible-ai-practices/. 
5 https://cloud.google.com/explainable-ai. 

6 https://modelcards.withgoogle.com/about. 
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technical expertise about Al, insight into responsible approaches to Al, and advice on how to 
turn Al ethical principles into practice. 


The Role of Regulation 


As our CEO Sundar Pichai wrote” recently, smart government approaches to regulation will 
play an important role in building trust and ensuring that Al technology is used responsibly, 
while also encouraging innovation. Regulatory frameworks should consider safety, 
explainability, fairness, and accountability to promote development of the right technologies 
in the right ways. Governments should also take a proportionate, risk-based approach by 
balancing potential harms with the social and economic benefits that will be created by Al. 
One of the biggest risks related to Al might be that we unnecessarily hamper its use in areas 
where it is urgently needed. So any regulatory framework should be flexible enough to evolve 
with this dynamic technology space. 


Smart regulatory policy can provide broad guidance across many sectors while allowing for 
tailored riskcmanagement solutions for individual Al applications in specific contexts. For 
some Al uses, such as Al-powered medical devices, existing regulatory frameworks are good 
starting points. In other instances, governments may need to update regulations or even 
create new regulatory frameworks. In all cases, regulators must take account of relevant 
costs and benefits and consider non-regulatory responses. There is no reasonable or 
practical “one-size-fits-all” approach. 


Our Comments on the Draft Memorandum 


OMB's draft principles (outlined below) represent a good foundation for advancing Al 
innovation while also protecting vital public interests. The principles will likewise help prevent 
conflicting regulatory approaches to Al from developing across the federal government. 
Going forward, we encourage OMB to develop supplemental guidance for regulatory 
agencies as experience leads to best practices on regulatory design and impact assessment, 
and to promote interagency discussions around key learnings. We also support efforts to 
ensure that agencies have adequate resources and in-house technical expertise to address Al 
as it continues to evolve. 


1. Public Trustin AI 


Google supports OMB's efforts to bolster public trust in Al. Indeed, this is a core objective 
that should unite technologists, businesses, policymakers, and citizens. The potential of this 
technology will not be fully realized if its development is held back by unfounded fears anq 
misunderstanding of Al. Public trust is best achieved if Al technology is developed responsibly 
and transparentiv. Accordingly, our Al principles encourage architectures with privacy 
safeguards, sufficient transparency and accountability over the use of data, and appropriate 
human direction and control. Google urges agencies to contribute to public efforts to 
demystify Al technology and highlight its societal benefits. 


7 https://www.ft.com/content/3467659a-386d-11ea-ac3c-f68c10993b04. 
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2. Public Participation 


Rulemaking related to Al technologies should include robust and fulsome opportunities for 
public participation and comment, which will allow diverse perspectives to be incorporated 
and help to avoid unintended consequences. These opportunities should begin at the earliest 
stages of the rulemaking and continue after the notice is published. Comment periods should 
be sufficiently long to allow stakeholders of all sizes to prepare useful comments, and 
supplemental comment should be requested where important questions are raised that 
warrant further consideration. Agencies should go above and beyond in terms of stakeholder 
outreach and engagement, including consideration of creative deep-dive formats (e.g.“citizen 
juries"5), expert roundtables, and more. To maximize opportunities for public participation, we 
encourage OMB to add additional language in the memo to this effect. 


3. Scientific Integrity and Information Quality 


Google agrees that Al policy decisions must be made based on scientifically sound analysis 
and the highest quality information. Technological innovation is rooted in the scientific 
method and a commitment to open inquiry, intellectual rigor, integrity, and collaboration. That 
is why our own AI principles emphasize the importance of aspiring to high standards of 
scientific excellence, including drawing on scientifically rigorous and multidisciplinary 
approaches. 


For example, when a research team develops a ML model for an application, such as 
automated lip reading, and writes a paper documenting the model's unprecedented accuracy 
and the team's evaluation process, we recommend a review aligning the research with 
potential socially beneficial uses and the estimated scale and severity of potential harms 
before sharing the paper's findings via publication or presentation. In the case of a lip-reading 
Al application, for example, the review could determine that the model has benefits for people 
with hearing or speech impairments and therefore is socially beneficial; the same review could 
also determine a potential harm such as use of the lip-reading model for nefarious 
surveillance purposes. They could apply frameworks such as international human rights to 
assess trade-offs, identify mitigations, and determine whether the model's benefits outweigh 
its harms, before deciding to share the research. 


Accordingly, Google encourages OMB to emphasize that Al knowledge should be shared 
responsibly, where practicable, through publication of educational materials, best practices, 
and research, as well as, when appropriate, free open-source tools that can enable more 
people to develop useful Al applications the right way. OMB should also urge agencies to 
invest in information quality for any open datasets released to the public for the purpose of 
training Al systems. 


4. Risk Assessment and Management 
Agencies must take a holistic, risk-based approach to Al. Like all other technologies, Al is not 
risk-free. But one of Al's greatest promises is that it will help reduce a vast array of risks 


inherent in everyday life, from traffic accidents and injuries sustained on the job, to 


8 https://www.epa.gov/international-cooperation/public-participation-guide-citizen-juries. 
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life-threatening illnesses, to the secondary effects related to the emission of pollutants. In 
assessing Al technology, reviewers should consider the risk-substitution impacts of a specific 
Al application against the non-use scenario. For example, commercial drones equipped with 
Al-powered analytics can be used to improve inspections of critical infrastructure, such as 
wind turbines mounted hundreds of feet in the air. Here we see the power of Al applications 
to advance social good — economic efficiency, environmental benefits, and safety. This is risk 
substitution in action, as one considers the human safety benefits of replacing human 
inspections with Al/drone-based inspections (see video example here'), as well as the 
opportunity costs of non-use. We urge agencies to be transparent and consistent as they 
weigh such considerations to ensure that innovation is not hampered by regulatory 
uncertainty or overreach. 


5. Benefits and Costs 


As noted above, Google agrees that cost-benefit analysis must be undertaken thoughtfully. 
Advances in Al will have transformative impacts in a wide range of fields, including healthcare, 
security, agriculture, energy, transportation, manufacturing, and entertainment. As agencies 
consider potential responses to Al technologies, they should be careful to take into account 
the broad range of social and economic factors and benefits related to those technologies, 
not solely a specific application's immediate impacts or hypothetical downsides. In this 
section of the memo, OMB should point agencies to the 'Non-Regulatorv Approaches to AI" 
section, urging them to consider novel approaches to understanding the impact of a given 
application before attempting to regulate it. This should include pilot programs and 
experiments, which will provide agencies with additional information to make an informed 
decision, rather than evaluating risks and costs based only on speculative ex ante forecasts of 
an application's use. 


ó. Flexibility 


AI technologies will allow us to make significant — and fast — safety, efficiency, and 
productivity advances in industry and throughout society. Google thus agrees that any 
government responses must be flexible in nature, not rigid or overly prescriptive. Because Al 
is a continuously evolving technology space, regulations should be designed to accommodate 
future innovation. A challenge for regulators will be to draft definitions that are sufficiently 
flexible to account for this inevitable change without being so vague and overbroad as to 
inject unnecessary uncertainty. OMB should consider how U.S. and international 
standard-setting bodies can be useful partners with government to assist in designing flexible 
regulatory and non-regulatory approaches and supporting their evolution over time. 


7. Fairness and Non-Discrimination 
Al algorithms and datasets can reflect, reinforce and amplify, or reduce unfair biases and 
discrimination. We recognize that defining unfair bias is not always simple, and notions of 


fairness differ across cultures and societies. Fairness is often multidimensional, and 
optimizing for one measure of fairness may require trading off another measure of fairness. 


9 https://www.youtube.com/watch?v=IF-u7j1x0C0&feature=youtu.be. 
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Google's Al principles highlight the need to avoid unjust impacts on people, particularly those 
related to sensitive characteristics such as race, ethnicity, gender, nationality, income, sexual 
orientation, ability, and political or religious beliefs. ML fairness is an emerging area of Al 
research in which we are heavily invested, and we have launched relevant open-source tools, 
including a What-If Tool? that empowers developers to visualize biases, Fairness Indicators" 
that help Cloud users check ML model performance against defined fairness metrics, and an 
ML Fairness Gym” for building model simulations that explore the potential long-run impacts 
of ML-based decision systems in social environments. 


We applaud OMB for ensuring that this is a focus and a priority. Moreover, OMB should 
consider urging agencies to find ways to promote best practices in this space, rather than 
merely assessing Al technologies on the basis of fairness and non-discrimination criteria. In 
this regard, organizations in the private sector and academia can be helpful partners. 


8. Disclosure and Transparency 


Google agrees that transparency is important, as we have highlighted prominently in our own 
Al principles. Google has pioneered the research? and implementation" of Model Cards, a 
framework for providing practical information about the performance and limitations of Al 
models in order to help developers make better decisions about what models to use for what 
purpose and how to deploy them responsibly. 


However, it is important for agencies to recognize that what constitutes appropriate 
disclosure and transparency will vary by industry, application, and even specific use cases and 
audiences. What's more, agencies must consider whether and how transparency features in 
certain circumstances conflict with privacy interests. Accordingly, we urge agencies to 
ensure that any government actions related to disclosure and transparency take a 
common-sense approach, are context-specific, and are not overly broad. 


9. Safety and Security 


We strongly believe that Al applications should be built and tested for safety. This isa 
foundational element of Google's Al principles, and we apply strong safety and security 
practices to avoid unintended results that create risks of harm. We also design our Al systems 
to be appropriately cautious and seek to develop them in accordance with best practices in Al 
safety research. In many cases, we will test Al technologies in constrained environments and 
monitor their operation after deployment. 


We urge agencies to consult widely with the private sector as questions of safety and security 
are explored and to stay up-to-date on emerging practices. This is another area where novel, 
non-regulatory approaches to Al applications must be considered. In this section, OMB 
should reinforce that agencies examine public-private collaboration (e.g., hackathons, 


10 https://pair-code. github.io/what-if-tool/. 

11 https://ai.googleblog.com/2019/12/fairness-indicators-scalable.html. 

12 https://ai.googleblog.com/2020/02/ml-fairness-gym-tool-for-exploring-long.html. 
13 https://arxiv.org/abs/1810.03993. 

14 https://modelcards.withgoogle.com/about. 
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challenges, and joint experiments) to promote safety and security. It is also an area ripe for 
partnership with U.S. and international industry standard-setting organizations (e.g., NIST, ISO, 
IEEE), consistent with the principles and policies set forth in OMB Circular A-119."° 


10. Interagency Coordination 


Google supports robust interagency coordination to prevent piecemeal and inconsistent 
government actions related to the development and use of Al applications. This should 
include fora and processes to allow agencies to share experience, learnings, and technical and 
regulatory know-how and capacity. Likewise, we strongly urge OMB to prioritize international 
coordination related to Al-related regulatory and non-regulatory actions, consistent with E.O. 
13609. The United States should strive with other nations that share our democratic and 
societal values to align regulatory approaches so that innovation is not held back by a 
confusing global patchwork of variable Al standards. The OECD Principles on Al” are an 
example of how countries and civil society can work together to reach consensus in this 
space. 


Non-Regulatory Approaches to AI 


As noted above, Google supports the OMB memorandum's direction that agencies examine 
non-regulatory approaches to Al, such as voluntary consensus standards, pilot programs, 
sandboxes, and sector-specific policy guidance. Al is a dynamic, quickly evolving technology 
space that will impact every sector of society. To strike the right balance, agencies should 
carefully consider, in the specific context of a particular Al application, options aside from a 
regulatory action. In some cases, this may warrant close monitoring to see how a particular 
application evolves and to further evaluate its societal impacts. In all of these circumstances, 
various non-regulatory approaches may be valuable tools to simultaneously promote 
adequate protections while also preserving the benefits of open innovation. They also offer 
the opportunity to harness the expertise of private sector technologists and to identify 
previously unforeseen opportunities for driving responsible innovation. 


Reducing Barriers to the Deployment and Use of AI 


Google strongly encourages reduction of barriers related to the responsible deployment and 
use of Al. OMB's emphasis on access to federal data is important. Consistent with legal, 
privacy, and security considerations, we urge agencies to proactively identify new 
opportunities for opening datasets to the public for use in development of Al applications. It 
is essential that federal agencies lead by example in adopting AI solutions that generate value 
for society and improve the functioning of government. 


Facilitating adoption of Al, however, extends beyond immediate regulatory considerations. It 
should be considered within the broader context of the Administration's work in areas such as 


15 https://www. mhitehouse.gov/wp-content/uploads/2017/11/Circular-119-1.pdf. 


16 


https://www.whitehouse.gov/sites/whitehouse.gov/files/omb/inforeg/inforeg/eo 13609/eo13609 05012012 
.pdf. 
17 https://www.oecd .org/going-digital/ai/principles/. 
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IT modernization, including migration from legacy infrastructure to the cloud. Reducing 
barriers for Al thus entails building bridges and encouraging adoption of new technology and 
IT infrastructure, without which Al cannot be deployed at scale in a meaningful way. 


Agency Plans to Achieve Consistency with this Memorandum 


Executive Order 13859 requires that implementing agencies submit plans to OMB for 
achieving consistency with the memorandum. Google urges OMB to ensure that agencies 
conduct outside consultations on their implementation plans, including with the public, 
business, and all other interested stakeholders. Moreover, we request that agencies publish 
and seek public comment before finalizing those individual agency plans. 


kkk 


Google appreciates the opportunity to comment on the memorandum and welcomes any 
questions, feedback, or opportunities for further discussion. 


Respectfully submitted, 


A tuf ll iz 


Kent Walker 
Senior Vice President, Global Affairs 
Google 


CO: 

Mr. Paul Ray 

Administrator 

Office of Information and Regulatory Affairs 


Mr. Dominic Mancini 
Deputy Administrator 
Office of Information and Regulatory Affairs 


Mr. Michael Kratsios 
Chief Technology Officer 
Office of Science and Technology Policy 


Ms. Lynne Parker 


Deputy Chief Technology Officer 
Office of Science and Technology Policy 
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FW: [EXTERNAL] Re: COVID-19 Al Dataset 


From «Parker, Lynne E. EOP/OSTP" Be 


"Kratsios, Michael J. EOP/OSTP" », "Hernandez, Elena R. 
To: EOP/OSTP' 'Calcagno, Drew A. EOP/OSTP' 


Ce: "Parker, Lynne E. EOP/OSTF' On 


Date: Fri, 13 Mar 2020 07:22:34 -0400 


From: Devvret Rishi 
Sent: Friday, March 13, 2020 4:01 AM 


To: Anthony Goldbloor ПІНУ 
Cc: Sebastian Kohlmeier ХК Parker, Lynne E. EOP/OSTP 


; Kratsios, Michael J. EOP/OSTP (Be 
Ben Hamner > 


Subject: Re: [EXTERNAL] Re: COVID-19 Al Dataset 


Thank you again very much for connecting today and providing a first draft of the research 
questions you are interested in. 


As Anthony mentioned, I wanted to send a prototype of what the data hosting & research 
questions may look like on Kaggle. Please note that all content is simply a placeholder right now, 
and will be updated once we have the finalized questions & dataset. We (and you) can also easily 
dynamically make updates as needed. 


1. You can access the dataset placeholder page using this link. 
2. You can then click the 'Tasks' button towards the top to see the list of open research 
questions for our ML community to answer. 
3. You can then click on individual tasks to read more, edit descriptions and see any 
submissions. 
I've also included a quick screen recording here that can show you the flow of seeing the dataset, 
the relevant questions (called Tasks on our platform) and placeholder submissions. 
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As Anthony mentioned, once we are ready to release this to the public we can also do more work 
to promote this dataset & questions including a potential prize for strong solutions. Look forward 
to working with you on this and please let me know if you have any questions. 


Thanks, 
Dev 


On Thu, Mar 12, 2020 at 6:03 PM Anthony Goldbloom BOTT 


wrote: 


Lynne, thanks for sharing the questions. Sebastian, nice to connect and to work with you. 


Dev cc'd is the owner on our side. As a next step, he's going to share the prototype page (target 
tonight). 


Sebastian, agree it makes sense for us to cross reference each other's websites (our community 
might like using Semantic Scholar as a tool for exploring the data). As a next step, can you 
and Dev connect tomorrow to figure out the best way for Kaggle to get dumps of the raw data? 
We have an API, so ideal would be something programmatic. 


On Thu, Mar 12, 2020 at 4:08 PM Sebastian Kohlmeier {BGI wrote: 


Hi Anthony, 


Great to meet you and fantastic to see Kaggle involved in this important effort! From a 
logistics perspective we plan to publish the dataset along with a description and associated 
metadata (including licensing information) on this web page (currently just a placeholder 
until we release the new content). Since Kaggle will be hosting the questions it will 
probably be easiest for us to cross-link from Semantic Scholar to Kaggle and vice-versa for 
Monday's release. If that makes sense, it would be great if you could connect me with 
someone at Kaggle so that we can coordinate tomorrow and ensure that the right links and 
content are in place. 


Thanks again and excited to see this go live on Monday! 
Sebastian Kohlmeier 
Sr. Manager - Business Operations | Semantic Scholar 


Allen Institute for AI (А12) 


On Thu, Mar 12, 2020 at 3:46 PM Parker, Lynne E. EOP/OSTP 
wrote: 


Hi Anthony, et al, 
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Please find attached a draft set of questions for the Kaggle site. Please treat this as a draft, for 
the purposes of your creating a prototype site, and don't distribute until we give you the final 
version. We recognize that this is a long set of questions, but I’m of the opinion that in this 
current crisis, the more different lines of work that we offer to researchers, the more benefits 
we might gain. If this presents an unwieldy setup for Kaggle, however, then let me know and we 
can prune the list (but recognizing that prioritizing which questions are potentially most 
impactful is very difficult for us to do). 


I’m also copying Sebastian Kohlmeier from AllenAl, who can talk with you about the dataset. Му 
understanding is that AllenAl would like to host the data, but would be fine with Kaggle 


mirroring it. 


We would like to go live with this effort on Monday, rather than tomorrow. When do you think 
you might have a draft site ready for us to review? 


Let me know if you need anything else just now. We very much appreciate your help and 
contribution to this effort! 


Lynne 
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Re: [EXTERNAL] Re: COVID-19 Al Dataset 


From Devuret Rishi (Bumi 
To: Sebastian Kohlmeier (bX6) . | |  .. 


Anthony Goldbloom , "Parker, Lynne E. EOP/OSTP" 
Cc: , Kratsios, Michael J. EOP/OSTP" 
‚ Ben Hamner (БЕН, 


Date: Fri, 13 Mar 2020 04:03:31 -0400 


Thank you Sebastian! Look forward to catching up. 

On Thu, Mar 12, 2020 at 8:03 PM Sebastian Kon Й wrote: 
Sounds good! Dev, I will schedule some time for us to sync tomorrow so that we can coordinate! 
Thanks again! 
Sebastian 


On Thu, Mar 12, 2020 at 6:03 PM Anthony Goldbloom (B (6) mmm". wrote: 
Lynne, thanks for sharing the questions. Sebastian, nice to connect and to work with you. 


Dev сс'а is the owner on our side. As a next step, he's going to share the prototype page (target tonight). 


Sebastian, agree it makes sense for us to cross reference each other's websites (our community might like using 
Semantic Scholar as a tool for exploring the data). As a next step, can you and Dev connect tomorrow to figure 
out the best way for Kaggle to get dumps of the raw data? We have an API, so ideal would be something 
programmatic. 


On Thu, Mar 12, 2020 at 4:08 PM Sebastian Kohlmeier (o wrote: 


Hi Anthony, 


Great to meet you and fantastic to see Kaggle involved in this important effort! From a logistics perspective 
we plan to publish the dataset along with a description and associated metadata (including licensing 
information) on this web page (currently just a placeholder until we release the new content). Since Kaggle 
will be hosting the questions it will probably be easiest for us to cross-link from Semantic Scholar to Kaggle 
and vice-versa for Monday's release. If that makes sense, it would be great if you could connect me with 
someone at Kaggle so that we can coordinate tomorrow and ensure that the right links and content are in 
place. 


Thanks again and excited to see this go live on Monday! 
Sebastian Kohlmeier 


Sr. Manager - Business Operations | Semantic Scholar 
Allen Institute for AI (AI2) 


On Thu, Маг 12, 2020 at 3:46 PM Parker, Lynne E. EOP/OSTP(BI (6 wrote: 


Hi Anthony, et al, 
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Please find attached a draft set of questions for the Kaggle site. Please treat this as a draft, for 
the purposes of your creating a prototype site, and don't distribute until we give you the final 
version. We recognize that this is a long set of questions, but I’m of the opinion that in this 
current crisis, the more different lines of work that we offer to researchers, the more benefits 
we might gain. If this presents an unwieldy setup for Kaggle, however, then let me know and 
we can prune the list (but recognizing that prioritizing which questions are potentially most 
impactful is very difficult for us to do). 


I'm also copying Sebastian Kohlmeier from AllenAl, who can talk with you about the dataset. 
My understanding is that AllenAl would like to host the data, but would be fine with Kaggle 
mirroring it. 


We would like to go live with this effort on Monday, rather than tomorrow. When do you think 
you might have a draft site ready for us to review? 


Let me know if you need anything else just now. We very much appreciate your help and 
contribution to this effort! 


Lynne 


Sebastian Kohlmeier 


Sr. Manager — Business Operations | Semantic Scholar 


Allen Institute - AI (А12) 


epic.org 
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RE: Call on Sunday 
From: "Parker, Lynne E. EOP/OSTP"'iBWWVWa а 


To: "Kratsios, Michael J. EOP/OSTP" , "Biggerstaff, 
x Matthew (CDC/DDID/NCIRD/ID" 

бех "Wilson, Matt B. EOP/OSTP" "Parker, Lynne E. 
| EOP/OSTP" 


Date: Fri, 13 Mar 2020 19:28:55 -0400 
Attachment COVID-19 Tech Task Рогов, White House Overview.pdf (69.3 kB) 
Hi Matt, 


Lynne 


Lynne E. Parker, Ph.D. 
Deputy Chief Technology Officer of the United States 
The White House 


Office: БШШ 
From: Kratsios, Michael J. EOP/OSTP (B6 


Sent: Friday, March 13, 2020 7:16 PM 
To: Biggerstaff, Matthew (CDC/DDID/NCIRD/I 
Cc: Parker, Lynne E. EOP/OSTP 


Wilson, Matt B. EOP/OSTP 


Subject: Call on Sunday 


Hi Matt — a group of high profile tech companies has self-assembled to work on better utilizing data to 
fight the crisis. Their next call is on Sunday. Lynne can send you the details. 


Many thanks! 
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Michael 


Michael Kratsios 
Chief Technology Officer of the United States 
The White House 
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FW: [EXTERNAL] Google comments on AI Regulatory 


Principles 


From: "Parker, Lynne E. EOP/OSTP" [BU 

To: "Kratsios, Michael J. EOP/OSTP" (DO 
Cc: "Parker, Lynne E. EOP/OSTP" (BUG) 
Date: Fri, 13 Mar 2020 15:33:09 -0400 


Attachment Google Submission - Draft OMB Memo - Guidance for Regulation of Artificial 
S: Intelligence Applications.pdf (171.05 kB) 


From: Miles Taylor 
Sent: Friday, March 13, 2020 12:48 PM 
To: Parker, Lynne E. EOP/OSTP (Be 


Cc: Michael Fitzpatrick (BG); Johanna Shelton (B6) 


Subject: [EXTERNAL] Google comments on AI Regulatory Principles 
Lynne, 


I wanted to let you know that Google has submitted comments (attached, which will also be 
posted here) on the OMB AI regulatory principles. | think you'll find them to be favorable 
toward the Administration's draft memo, along with some constructive suggestions. As 
you and | have discussed, we think the framework strikes the right balance, and we're 
grateful for your team's leadership in this space. Let me know if you all have any 
questions. We look forward to engaging on this further! 


Best, M 
[=] Miles Taylor 
Government Affairs & Public 
Policy 
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Google 


March 12, 2020 


Mr. Russell Vought 

Acting Director 

Office of Management and Budget 
Washington, DC 20503 


Dear Mr. Vought, 


Google welcomes the opportunity to provide comments in response to the Office of 
Management and Budget's Request for Comments on a Draft Memorandum to the Heads of 
Executive Departments and Agencies, “Guidance for Regulation of Artificial Intelligence 
Applications.” In particular, we support the memorandum's message that artificial 
intelligence (Al) and machine learning (ML) hold extraordinary potential to improve our lives 
and that the responsible development of Al technology should be encouraged, not 
discouraged. Moreover, we agree that federal agencies must avoid creating unnecessary 
barriers to Al development and use by taking a thoughtful, context-specific approach when 
regulating Al that carefully assesses risks and weighs costs and benefits. 


The Promise of Al 


Google's mission is to organize the world's information and make it universally accessible and 
useful. In pursuing this mission, Google is prioritizing investment in advanced technologies 
such as AI/ML. These technologies make Google's core products and services much more 
useful to the public, including Android, Assistant, Cloud, Gmail, Maps, Photos, Pixel, Search, 
YouTube, and many more. 


Google is also creating tools to ensure that everyone can access Al, including researchers and 
developers, entrepreneurs and businesses of all sizes, academics, nonprofits, and 
governments. Wider accessibility is how Al will have its biggest impact and how society can 
reap its full promise. Critical to this approach is open-sourcing Al tools through systems such 
as TensorFlow, a framework which makes machine learning faster, smarter, more flexible, and 
available to a wider community of developers. Google Cloud brings this technology to the 
enterprise world, offering a range of Al-powered products and solutions, from pre-built APIs 
for computer vision and naturallanguage processing to end-to-end solutions that are helping 
to transform sectors such as financial services, retail, healthcare, and beyond. 


Al is making it easier for people to go about their daily lives, from managing household and 
workplace tasks to breaking down language barriers, while also transforming organizations 
across both the private and public sectors. But the future potential is far greater. Al provides 
new hope for addressing the world's toughest problems, from rethinking transportation to 


1 Federal Register Doc. 2020-00261, 
https://www.whitehouse.gov/wp-content/uploads/2020/01/Draft-OMB-Memo-on-Regulation-of-Al-1-7-19.p 
df. 

2 https://www.tensorflow.org/. 
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advancing scientific discovery to tackling complex environmental challenges. That's why 
we're conducting research that advances the state-of-the-art in the field, applying Al across 
many domains. For instance, we've recently pioneered an Al model that can help doctors spot 
breast cancer in mammograms with greater accuracy, and another that makes immediate, 
hyperlocal rainfall forecasts more accurately than existing methods to help everything from 
agricultural productivity to resilience against flooding. 


Using Al Responsibly 


We believe that Al willbe overwhelmingly beneficial for society. But it is critical that the 
technology is developed and used to help people — that it is socially beneficial, fair, 
accountable, and works for everyone. Al must be developed and used responsibly in ways 
that build trust. And potential harms must be acknowledged and mitigated, or prevented. 


As a leader in the development and use of Al, Google has recognized both the enormous 
benefits and the issues that Al raises, and has wrestled with those issues in the context of our 
own operations. To that end, in 2018 we established principles governing our development 
and use of Google А! applications, best practices to share in our work with communities 
outside of Google, and programs to operationalize our efforts. 


Our Al Principles? guide the ethical development and use of Al in our research, products, and 
services. These guidelines help us avoid unfair bias, rigorously review for safety, design with 
privacy top-of-mind, and make the technology accountable to people. They also specify 
areas where we will not design or deploy Al, such as where human rights might be negatively 
impacted. But principles that simply remain on paper are meaningless. So we've developed 
business processes to put them into action, such as requiring vigorous testing of Google's AI 
decisions for fairness and conducting independent assessments of new products against our 
principles. 


We have gone even further and are leading efforts to push for the responsible and socially 
beneficial use of Al applications developed outside of Google. We believe that all 
organizations creating Al tools should adopt guiding principles and robust internal review 
processes. That is why we regularly share our recommended responsible Al practices^ with 
the broader Al community, other companies, nonprofit organizations, and academia. Google 
Cloud has also created commercial tools that enable enterprise organizations to incorporate 
these practices into their own Al applications. For example, we have introduced tools like 
Explainable Al? to help our customers better test and understand the outputs of their models. 
Additionally, we have invested in scaling frameworks like Model Cards' to increase 
transparency and comprehension around the proper use and limitations of Al models. 


What's more, we've worked closely with governments and civil society stakeholders around 
the world to seize opportunities and address challenges related to Al. This includes 
stakeholders at all levels of government here in the United States. Google has provided 


3 https://ai.google/principles/. 

^ https://ai.google/responsibilities/responsible-ai-practices/. 
5 https://cloud.google.com/explainable-ai. 

6 https://modelcards.withgoogle.com/about. 
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technical expertise about Al, insight into responsible approaches to Al, and advice on how to 
turn Al ethical principles into practice. 


The Role of Regulation 


As our CEO Sundar Pichai wrote” recently, smart government approaches to regulation will 
play an important role in building trust and ensuring that Al technology is used responsibly, 
while also encouraging innovation. Regulatory frameworks should consider safety, 
explainability, fairness, and accountability to promote development of the right technologies 
in the right ways. Governments should also take a proportionate, risk-based approach by 
balancing potential harms with the social and economic benefits that will be created by Al. 
One of the biggest risks related to Al might be that we unnecessarily hamper its use in areas 
where it is urgently needed. So any regulatory framework should be flexible enough to evolve 
with this dynamic technology space. 


Smart regulatory policy can provide broad guidance across many sectors while allowing for 
tailored riskcmanagement solutions for individual Al applications in specific contexts. For 
some Al uses, such as Al-powered medical devices, existing regulatory frameworks are good 
starting points. In other instances, governments may need to update regulations or even 
create new regulatory frameworks. In all cases, regulators must take account of relevant 
costs and benefits and consider non-regulatory responses. There is no reasonable or 
practical “one-size-fits-all” approach. 


Our Comments on the Draft Memorandum 


OMB's draft principles (outlined below) represent a good foundation for advancing Al 
innovation while also protecting vital public interests. The principles will likewise help prevent 
conflicting regulatory approaches to Al from developing across the federal government. 
Going forward, we encourage OMB to develop supplemental guidance for regulatory 
agencies as experience leads to best practices on regulatory design and impact assessment, 
and to promote interagency discussions around key learnings. We also support efforts to 
ensure that agencies have adequate resources and in-house technical expertise to address Al 
as it continues to evolve. 


1. Public Trustin AI 


Google supports OMB's efforts to bolster public trust in Al. Indeed, this is a core objective 
that should unite technologists, businesses, policymakers, and citizens. The potential of this 
technology will not be fully realized if its development is held back by unfounded fears anq 
misunderstanding of Al. Public trust is best achieved if Al technology is developed responsibly 
and transparentiv. Accordingly, our Al principles encourage architectures with privacy 
safeguards, sufficient transparency and accountability over the use of data, and appropriate 
human direction and control. Google urges agencies to contribute to public efforts to 
demystify Al technology and highlight its societal benefits. 


7 https://www.ft.com/content/3467659a-386d-11ea-ac3c-f68c10993b04. 
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2. Public Participation 


Rulemaking related to Al technologies should include robust and fulsome opportunities for 
public participation and comment, which will allow diverse perspectives to be incorporated 
and help to avoid unintended consequences. These opportunities should begin at the earliest 
stages of the rulemaking and continue after the notice is published. Comment periods should 
be sufficiently long to allow stakeholders of all sizes to prepare useful comments, and 
supplemental comment should be requested where important questions are raised that 
warrant further consideration. Agencies should go above and beyond in terms of stakeholder 
outreach and engagement, including consideration of creative deep-dive formats (e.g.“citizen 
juries"5), expert roundtables, and more. To maximize opportunities for public participation, we 
encourage OMB to add additional language in the memo to this effect. 


3. Scientific Integrity and Information Quality 


Google agrees that Al policy decisions must be made based on scientifically sound analysis 
and the highest quality information. Technological innovation is rooted in the scientific 
method and a commitment to open inquiry, intellectual rigor, integrity, and collaboration. That 
is why our own AI principles emphasize the importance of aspiring to high standards of 
scientific excellence, including drawing on scientifically rigorous and multidisciplinary 
approaches. 


For example, when a research team develops a ML model for an application, such as 
automated lip reading, and writes a paper documenting the model's unprecedented accuracy 
and the team's evaluation process, we recommend a review aligning the research with 
potential socially beneficial uses and the estimated scale and severity of potential harms 
before sharing the paper's findings via publication or presentation. In the case of a lip-reading 
Al application, for example, the review could determine that the model has benefits for people 
with hearing or speech impairments and therefore is socially beneficial; the same review could 
also determine a potential harm such as use of the lip-reading model for nefarious 
surveillance purposes. They could apply frameworks such as international human rights to 
assess trade-offs, identify mitigations, and determine whether the model's benefits outweigh 
its harms, before deciding to share the research. 


Accordingly, Google encourages OMB to emphasize that Al knowledge should be shared 
responsibly, where practicable, through publication of educational materials, best practices, 
and research, as well as, when appropriate, free open-source tools that can enable more 
people to develop useful Al applications the right way. OMB should also urge agencies to 
invest in information quality for any open datasets released to the public for the purpose of 
training Al systems. 


4. Risk Assessment and Management 
Agencies must take a holistic, risk-based approach to Al. Like all other technologies, Al is not 
risk-free. But one of Al's greatest promises is that it will help reduce a vast array of risks 


inherent in everyday life, from traffic accidents and injuries sustained on the job, to 


8 https://www.epa.gov/international-cooperation/public-participation-guide-citizen-juries. 
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life-threatening illnesses, to the secondary effects related to the emission of pollutants. In 
assessing Al technology, reviewers should consider the risk-substitution impacts of a specific 
Al application against the non-use scenario. For example, commercial drones equipped with 
Al-powered analytics can be used to improve inspections of critical infrastructure, such as 
wind turbines mounted hundreds of feet in the air. Here we see the power of Al applications 
to advance social good — economic efficiency, environmental benefits, and safety. This is risk 
substitution in action, as one considers the human safety benefits of replacing human 
inspections with Al/drone-based inspections (see video example here'), as well as the 
opportunity costs of non-use. We urge agencies to be transparent and consistent as they 
weigh such considerations to ensure that innovation is not hampered by regulatory 
uncertainty or overreach. 


5. Benefits and Costs 


As noted above, Google agrees that cost-benefit analysis must be undertaken thoughtfully. 
Advances in Al will have transformative impacts in a wide range of fields, including healthcare, 
security, agriculture, energy, transportation, manufacturing, and entertainment. As agencies 
consider potential responses to Al technologies, they should be careful to take into account 
the broad range of social and economic factors and benefits related to those technologies, 
not solely a specific application's immediate impacts or hypothetical downsides. In this 
section of the memo, OMB should point agencies to the 'Non-Regulatorv Approaches to AI" 
section, urging them to consider novel approaches to understanding the impact of a given 
application before attempting to regulate it. This should include pilot programs and 
experiments, which will provide agencies with additional information to make an informed 
decision, rather than evaluating risks and costs based only on speculative ex ante forecasts of 
an application's use. 


ó. Flexibility 


AI technologies will allow us to make significant — and fast — safety, efficiency, and 
productivity advances in industry and throughout society. Google thus agrees that any 
government responses must be flexible in nature, not rigid or overly prescriptive. Because Al 
is a continuously evolving technology space, regulations should be designed to accommodate 
future innovation. A challenge for regulators will be to draft definitions that are sufficiently 
flexible to account for this inevitable change without being so vague and overbroad as to 
inject unnecessary uncertainty. OMB should consider how U.S. and international 
standard-setting bodies can be useful partners with government to assist in designing flexible 
regulatory and non-regulatory approaches and supporting their evolution over time. 


7. Fairness and Non-Discrimination 
Al algorithms and datasets can reflect, reinforce and amplify, or reduce unfair biases and 
discrimination. We recognize that defining unfair bias is not always simple, and notions of 


fairness differ across cultures and societies. Fairness is often multidimensional, and 
optimizing for one measure of fairness may require trading off another measure of fairness. 


9 https://www.youtube.com/watch?v=IF-u7j1x0C0&feature=youtu.be. 
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Google's Al principles highlight the need to avoid unjust impacts on people, particularly those 
related to sensitive characteristics such as race, ethnicity, gender, nationality, income, sexual 
orientation, ability, and political or religious beliefs. ML fairness is an emerging area of Al 
research in which we are heavily invested, and we have launched relevant open-source tools, 
including a What-If Tool? that empowers developers to visualize biases, Fairness Indicators" 
that help Cloud users check ML model performance against defined fairness metrics, and an 
ML Fairness Gym” for building model simulations that explore the potential long-run impacts 
of ML-based decision systems in social environments. 


We applaud OMB for ensuring that this is a focus and a priority. Moreover, OMB should 
consider urging agencies to find ways to promote best practices in this space, rather than 
merely assessing Al technologies on the basis of fairness and non-discrimination criteria. In 
this regard, organizations in the private sector and academia can be helpful partners. 


8. Disclosure and Transparency 


Google agrees that transparency is important, as we have highlighted prominently in our own 
Al principles. Google has pioneered the research? and implementation" of Model Cards, a 
framework for providing practical information about the performance and limitations of Al 
models in order to help developers make better decisions about what models to use for what 
purpose and how to deploy them responsibly. 


However, it is important for agencies to recognize that what constitutes appropriate 
disclosure and transparency will vary by industry, application, and even specific use cases and 
audiences. What's more, agencies must consider whether and how transparency features in 
certain circumstances conflict with privacy interests. Accordingly, we urge agencies to 
ensure that any government actions related to disclosure and transparency take a 
common-sense approach, are context-specific, and are not overly broad. 


9. Safety and Security 


We strongly believe that Al applications should be built and tested for safety. This isa 
foundational element of Google's Al principles, and we apply strong safety and security 
practices to avoid unintended results that create risks of harm. We also design our Al systems 
to be appropriately cautious and seek to develop them in accordance with best practices in Al 
safety research. In many cases, we will test Al technologies in constrained environments and 
monitor their operation after deployment. 


We urge agencies to consult widely with the private sector as questions of safety and security 
are explored and to stay up-to-date on emerging practices. This is another area where novel, 
non-regulatory approaches to Al applications must be considered. In this section, OMB 
should reinforce that agencies examine public-private collaboration (e.g., hackathons, 


10 https://pair-code. github.io/what-if-tool/. 

11 https://ai.googleblog.com/2019/12/fairness-indicators-scalable.html. 

12 https://ai.googleblog.com/2020/02/ml-fairness-gym-tool-for-exploring-long.html. 
13 https://arxiv.org/abs/1810.03993. 

14 https://modelcards.withgoogle.com/about. 
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challenges, and joint experiments) to promote safety and security. It is also an area ripe for 
partnership with U.S. and international industry standard-setting organizations (e.g., NIST, ISO, 
IEEE), consistent with the principles and policies set forth in OMB Circular A-119."° 


10. Interagency Coordination 


Google supports robust interagency coordination to prevent piecemeal and inconsistent 
government actions related to the development and use of Al applications. This should 
include fora and processes to allow agencies to share experience, learnings, and technical and 
regulatory know-how and capacity. Likewise, we strongly urge OMB to prioritize international 
coordination related to Al-related regulatory and non-regulatory actions, consistent with E.O. 
13609. The United States should strive with other nations that share our democratic and 
societal values to align regulatory approaches so that innovation is not held back by a 
confusing global patchwork of variable Al standards. The OECD Principles on Al” are an 
example of how countries and civil society can work together to reach consensus in this 
space. 


Non-Regulatory Approaches to AI 


As noted above, Google supports the OMB memorandum's direction that agencies examine 
non-regulatory approaches to Al, such as voluntary consensus standards, pilot programs, 
sandboxes, and sector-specific policy guidance. Al is a dynamic, quickly evolving technology 
space that will impact every sector of society. To strike the right balance, agencies should 
carefully consider, in the specific context of a particular Al application, options aside from a 
regulatory action. In some cases, this may warrant close monitoring to see how a particular 
application evolves and to further evaluate its societal impacts. In all of these circumstances, 
various non-regulatory approaches may be valuable tools to simultaneously promote 
adequate protections while also preserving the benefits of open innovation. They also offer 
the opportunity to harness the expertise of private sector technologists and to identify 
previously unforeseen opportunities for driving responsible innovation. 


Reducing Barriers to the Deployment and Use of AI 


Google strongly encourages reduction of barriers related to the responsible deployment and 
use of Al. OMB's emphasis on access to federal data is important. Consistent with legal, 
privacy, and security considerations, we urge agencies to proactively identify new 
opportunities for opening datasets to the public for use in development of Al applications. It 
is essential that federal agencies lead by example in adopting Al solutions that generate value 
for society and improve the functioning of government. 


Facilitating adoption of Al, however, extends beyond immediate regulatory considerations. It 
should be considered within the broader context of the Administration's work in areas such as 


15 https://www. mhitehouse.gov/wp-content/uploads/2017/11/Circular-119-1.pdf. 


16 


https://www.whitehouse.gov/sites/whitehouse.gov/files/omb/inforeg/inforeg/eo 13609/eo13609 05012012 
.pdf. 
17 https://www.oecd .org/going-digital/ai/principles/. 
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IT modernization, including migration from legacy infrastructure to the cloud. Reducing 
barriers for Al thus entails building bridges and encouraging adoption of new technology and 
IT infrastructure, without which Al cannot be deployed at scale in a meaningful way. 


Agency Plans to Achieve Consistency with this Memorandum 


Executive Order 13859 requires that implementing agencies submit plans to OMB for 
achieving consistency with the memorandum. Google urges OMB to ensure that agencies 
conduct outside consultations on their implementation plans, including with the public, 
business, and all other interested stakeholders. Moreover, we request that agencies publish 
and seek public comment before finalizing those individual agency plans. 


kkk 


Google appreciates the opportunity to comment on the memorandum and welcomes any 
questions, feedback, or opportunities for further discussion. 


Respectfully submitted, 


A tuf ll iz 


Kent Walker 
Senior Vice President, Global Affairs 
Google 


CO: 

Mr. Paul Ray 

Administrator 

Office of Information and Regulatory Affairs 


Mr. Dominic Mancini 
Deputy Administrator 
Office of Information and Regulatory Affairs 


Mr. Michael Kratsios 
Chief Technology Officer 
Office of Science and Technology Policy 


Ms. Lynne Parker 


Deputy Chief Technology Officer 
Office of Science and Technology Policy 
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Re: [EXTERNAL] Re: COVID-19 Open Research Dataset Press 
Call 


СТ Anthony Goldbioom (BS ERN 
то: Dewvet Fish NOTI 


"Hernandez, Elena R. EOP/OSTP" >, "Kratsios, Michael 
Cc: J. EOP/OSTP" Parker, Lynne E. EOP/OSTP" 
Pete Voss 


Date: Sat, 14 Mar 2020 23:38:36 -0400 


Elena, some small changes to my quote. 


"Its difficult for people to manually go through more than 20,000 articles and svnthesize their findings. Recent 
advances in technology can be helpful here. We're putting machine readable versions of these articles in front of our 
community of more than 4 million data scientists. Our hope is that AI can be used to help find answers to a key set 
of questions about COVID-19." 


Also adding Pete from Google's comms team. 


Anthony 


On Sat, Mar 14, 2020 at 3:50 PM Devvret Rishi (Of > wrote: 


Thank you Elena, Michael, and Lynne — really looking forward to this launch. 


The URL Re the dataset and challenges which will be made public on Monday at 2pm EST 
-for-ai/CORD-19-research-challenge<. We may make a 
short URL version ої this before launch on Monday, in which case I will also circulate that with you asap. 


In case you're interested, you can view a staging (not for public release) version of the dataset here and the list of 
the tasks here. We're reviewing these iteratively with Lynne. 


Best, 
Dev 


On Sat, Mar 14, 2020 at 3:38 PM Anthony «увы wrote: 


Elena, adding dev from our team. He can share the URLs with you (to replace the placeholders in the current 
press release). 


On Sat, Mar 14, 2020, 11:04 AM Hernandez, Elena R. EOP/OSTP BB E > wrote: 


That’s perfect, thanks Anthony for the quick turnaround! We are planning for 2pm on Monday for 
the call. 


Will be back in touch soon with more details on the call and the latest version of the press release 
for your awareness, but let us know if you have any questions in the meantime. | can also be 


reached at (WI 
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Elena 


Elena Hernandez 

Press Secretary 

Office of Science and Technology Policy 
The White House 


ү oic: (GEN, 
TON 


From: Anthonv Goldbloom 
Sent: Saturdav, March 14, 2020 1:56 PM 


To: Kratsios, Michael J. EOP/OSTP BMS 
Cc: Parker, Lynne E. EOP/OSTP (B (60 >; Hernandez, Elena В. EOP/OSTP 


Subject: [EXTERNAL] Re: COVID-19 Open Research Dataset Press Call 


Michael for sure. 


Lynne and Elena, as far as quotes are concerned, does this work? I'd plan to say something similar on the 
teleconference. 


"It's difficult for humans to manually go through 24K- articles and synthesize their findings. Recent 
advances in the ability of machines to process natural language mean that AI might be helpful here. We're 
putting this dataset up in front of our community of 4.3MM data scientists in the hope that the world's AI 
community can help find answers to a key set of questions about COVID- 19." 


Anthony 
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On Sat, Mar 14, 2020 at 8:36 AM Kratsios, Michael J. EOP/OSTP D- 


wrote: 


Hi Anthony, 


We are so grateful for the work by your team to prepare the dataset and tasks for Monday. Having it up on 
your platform will make a really big difference. 


To issue our call to action and raise awareness across both the public and the scientific community, the we 
are hosting a media briefing call on Monday and putting out a press release. I plan to give brief remarks 
and would love to have you dial-in and speak for a minute or two as well on kaggle's roll in this effort. 
Please let us know if you can join. 


Attached is a the draft press release and the comms plan. If you'd like to include a quote, feel free to send 
it to Lynne and Elena (cced) who are running point on the Monday roll-out. 


Thanks again for all your help! 


Best, 


Michael 


Michael Kratsios 

Deputy Assistant to the President 

Chief Technology Officer of the United States 
The White House 

0: 
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Re: [EXTERNAL] Re: COVID-19 Tech Response | Call 
tomorrow? 


From “Courtney Gregoire (CELA) (ENTI 
To: | Ісмібегізоп «ПИ». iter (Ete 


"Kratsios, Michael J. EOP/OSTP' >, "Fred Humphries 

H Johanna Shelton Lee Carosi 
"Parker, Lynne E. EOP/OSTP" 
, pavton , Portia Wu 


топіка “pickles (БКО >. 


Сс: 


Date: Sat, 14 Mar 2020 20:29:06 -0400 


Works for Microsoft as well 
From: Leslie Miller 
Sent: Saturdav, March 14, 2020 4:34 PM 


To: Iculbertson 
Cc: Kratsios, Michael J. EOP/OSTP; Fred Humphries (CELA); Johanna Shelton; Lee Carosi Dunn; Parker, Lvnne E. 


EOP/OSTP; pavton; Portia Wu; Courtnev Gregoire (CELA); monika; npickles; Tvler Otto 
Subject: [EXTERNAL] Re: COVID-19 Tech Response | Call tomorrow? 


That time works for VouTube/Google as well, thanks. 


On Sat, Mar 14, 2020 at 3:13 PM Lauren Culbertson SETT wrote: 


Works for Twitter. Thanksl 


On Sat, Mar 14, 2020 at 6:02 PM Kratsios, Michael J. EOP/OSTP (BIS wrote: 


Hi AII — wanted to follow-up on our call. Can we set up a time for Monday to reconvene and get an update on 
your current thinking? 


Say, 4pm? 


Thanks, 


Michael 


From: Kratsios, Michael J. EOP/OSTP 
Sent: Wednesday, March 11, 2020 6:16 PM 


Ce: 'Fred Humphries (СЕГА)! ; Portia W Payton 


Theme ; Johanna Shelton' ; Lee Carosi Dunn 
"Lauren Culbertson' < >; Parker, Lynne E. EOP/OSTP 
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Subject: COVID-19 Tech Response | Call tomorrow? 


Hi Courtnev, Monika, Nick, Leslie, and Tyler, 


At today's White House COVID-19 Tech Meeting (see attached for the readout), one of the important work 


streams we identified is for better coordination to address COVID-19 misinformation on online platforms. 


One industry collaboration model that might be useful is GIFCT, which you are all familiar with. I think you all 
could provide great insight into workable cross-industry mechanisms to facilitate information sharing, and to 


help us set up something similar for COVID-19. And do so very quickly. 


Are you available for a short call tomorrow, say noon eastern? We'd greatly appreciate leveraging your 


expertise on this challenge. 


Many thanks, 


Michael 


Michael Kratsios 

Deputy Assistant to the President 

Chief Technology Officer of the United States 
The White House 

o: 


#mobile 7 


Leslie Miller 
Google LLC 
1600 Amphitheatre Parkway 
Mountain View, CA 94043 

o 

c 
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RE: [EXTERNAL] Re: COVID-19 Al Dataset 


From "Parker, Lynne E. EOP/OSTP NETT 


To: 'Kratsios, Michael J. EOP/OSTP' 'Hemandez, Elena R. 
l EOP/OSTP' 


Cc: "Parker, Lynne E. EOP/OSTF' BOTT 


Date: Sat, 14 Mar 2020 20:27:51 -0400 


Thanks. EE wawww 


From: Kratsios, Michael J. EOP/OSTP 
Sent: Saturday, March 14, 2020 7:28 PM 


To: Parker, Lynne E. EOP/OSTPIBWOVEB€UO5EAINIIAIIAKNK4É Hernandez, Elena R. EOP/OSTP 


Subject: RE: [EXTERNAL] Re: COVID-19 AI Dataset 


From: Parker, Lynne E. EOP/OSTP Б ЕЕЕ 


Sent: Saturday, March 14, 2020 7:15 PM 


To: Kratsios, Michael J. EOP/OSTP (Be Hernandez, Elena R. EOP/OSTP 
(6) > 


Сс: Parker, Lynne E. EOP/OSTP 
Subject: FW: [EXTERNAL] Re: COVID-19 AI Dataset 
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From: Devvret Rishi 
Sent: Saturday, March 14, 2020 7:02 PM 


To: Parker, Lynne E. EOP/OSTP 4B) 


Cc: Anthony Goldbloom Sebastian 


“ША тш Т 


Subject: Ке: [EXTERNAL] Ве: COVID-19 Al Dataset 


Hi Lynne, 


We've made all the updates to the dataset and created tasks from the new questions from 
NASEM. They're ready for your review again from the shared link. 


One note on the Tasks — We've currently erred on the side of including all the questions we can, 
even those that may not be a good ftt for data mining. We're going to take another pass to see 
which we can consolidate further (and in some cases, discard if it's a particularly poor fit). Please 
let us know if you have any strong preferences on how we do this, or if you have specific Tasks 
you'd like to flag for edits or removal after your review. 


I wanted to also raise one other item for your consideration. In coordination with the Allen 
Institute, we've decided we'd like to offer monetary prizes to incentivize participation in the 
challenge. The budget for these will come from our team, and the prizes may take the form of 
donations to charitable organizations working on COVID-19 response (the latter point is pending 
confirmation on logistics). There are two items I would appreciate your feedback on here: 


e For the prizes, we'd like to have a timeline for eligibility. We plan to communicate that 
we'll do the first round of review of submissions about 6 weeks after launch, and close 
the prize window overall 3 months after launch. Do these timelines seem appropriate to 
you, or any flags you'd like to raise? The dataset, tasks and submission window will 
continue to be live after the prize window close. 

e [t would be great to have a subject-matter expert who can help us review the submissions 
and decide which does the best in addressing the underlying research questions, against a 
rubric we've set. We hope to work with the Allen Institute on this, but wanted to also see 
if you had any recommended judges to review the submissions. 

Thanks, 
Dev 


On Sat, Mar 14, 2020 at 8:57 AM Parker, Lynne E. EOP/OSTP 


GENN: wrote: 


Hi Dev, 
One more edit — please update the acknowledgements as follows: 


This dataset was created by the Allen Institute for Al in partnership with the Chan Zuckerberg 
Initiative, Georgetown University's Center for Security and Emerging Technology, Microsoft 
Research, and the National Library of Medicine - National Institutes of Health, in coordination with 
the White House Office of Science and Technology Policy. 
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Thanks, 
Lynne 


From: Parker, Lynne E. EOP/OSTP 
Sent: Saturday, March 14, 2020 10:45 AM 


To: 'Dewret Rishi' (B(8)] 
Cc: Anthony Goldbloom (GI Sebastian 
Kohlmeier (BGS; Ben Hamne BGE Julia Elliott 

; Parker, Lynne E. EOP/OSTP Se 


Subject: RE: [EXTERNAL] Re: COVID-19 Al Dataset 
Hi Dev, 
Updating the call to action: 


We are issuing a call to action to the world's artificial intelligence experts to develop text and data 
mining tools that can help the medical community develop answers to high priority scientific 
questions. The CORD-19 dataset represents the most extensive machine-readable coronavirus 
literature collection available for data mining to date. This allows the worldwide AI research 
community the opportunity to apply text and data mining approaches to find answers to questions 
within, and connect insights across, this content in support of the ongoing COVID-19 response 
efforts worldwide. There is a growing urgency for these approaches because of the rapid 
increase in coronavirus literature, making it difficult for the medical community to keep up. 


A list of our initial key questions can be found under the Tasks section of this dataset. These key 
scientific questions are drawn from the NASEM's SCIED (National Academies of Sciences, Engineering, 
and Medicine’s Standing Committee on Emerging Infectious Diseases and 21st Century Health 


Threats) research topics <link to >https://www.nationalacademies.org/event/03-11-2020/standing- 
committee-on-emerging-infectious-diseases-and-21st-century-health-threats-virtual-meeting-1<> 
and the World Health Organization’s R&D Blueprint for COVID-19 <link to 
>https://www.who.int/blueprint/priority-diseases/key- 


action/Global Research Forum FINAL VERSION for web 14 feb 2020.pdf?ua=1<>. Many of these 
questions are suitable for text mining, and we encourage researchers to develop text mining tools 
to provide insights on these questions. 


(1 see you also have the above acknowledgement on the tasks pages. It seems to me to work better 
on the landing page in the call to action, per above.) 


Throughout the questions, please capitalize 'COVID-19'. 
lIl review again once you have a chance to add the questions. 


Thanks again, 
Lynne 


From: Parker, Lynne E. EOP/OSTP (БУВ) i i s i; 


Sent: Saturday, March 14, 2020 10:02 AM 
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To: ‘Dewret Rishi БУКВУ 
Cc: Anthony Goldbloom Бб} ЕЕЕ: Sebastian 


Kohlmeier 


; Ben Hamner >; Julia Elliott 
Parker, Lynne E. EOP/OSTP > 


Subject: RE: [EXTERNAL] Re: COVID-19 AI Dataset 


Hi Dev, 


Many thanks. | will review. On your first point, f'Il talk with my WH colleagues and get back with 
you. l understand your point. 


I completely agree on bullet H2. Please work with Sebastian, et al, and feel free to beef up the 
description with content they suggest for the dataset description. 


Also, we have just received the second set of questions — this one from the NASEM's SCEID (National 
Academies of Science, Engineering, and Medicine's Standing Committee on Emerging Infectious 


Diseases and 21°t Century Health Threats), which are now public here: 


-committee-on-emerging- 


не diseases-and-21st-century-health-threats-virtual-meeting-1<. Please ада them to the 
questions you already have from WHO. 


As with the WHO questions, these are not framed for text mining. Taking a similar approach that you 
took for the WHO questions makes sense here, too. 


Also, we need to acknowledge the source of these questions. Per one of my earlier notes, at the end 
of the Call to Action (after the sentence that says “A list of our initial...", please add the following (OK 
to wordsmith if needed): 


"These key scientific questions are drawn from the NASEM's SCIED (National Academies of Sciences, 
Engineering, and Medicine's Standing Committee on Emerging Infectious Diseases and 21st Century 


Health Threats) research rn <link to >https: Lhwww. nationalacademies.org/event/03-11- 


meeting-1<> and the World Health Organization’s R&D „согын for COVID-19 <link to 
>https://www.who.int/blueprint/priority-diseases/key- 


action/Global Research Forum FINAL VERSION for web 14 feb 2020.pdf?ua=1<>.” 


Thanks, 
Lvnne 


From: Devvret Rishi 
Sent: Saturdav, March 14, 2020 3:52 AM 


To: Parker, Lynne E. EOP/osTP (BS 
Cc: Anthony Goldbloom ; Sebastian 
Kohlmeier »; Ben Hamner ; Julia Elliott 


Subject: Re: [EXTERNAL] Re: COVID-19 AI Dataset 
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Hi Lynne, 


The content is ready for your review. You can view the dataset using this link, and see the list 
of open research questions in the `Tasks` tab (or through this link). 


I also had two follow-up questions I wanted to get your thoughts on: 


e Anthony pointed out that being more clear that the White House has an interest in the 
outcome of the work done here could help make this more meaningful for our 
community and incentivize higher quality engagement. Would we have your 
permission to add in language such as "The White House has asked us to issue a call to 
action to the world's artificial intelligence..."? We can take a light touch but 
mentioning the involvement will likely have an impact. 

The current dataset description is certainly helpful, but it would be great if we could 
give some additional context on the underlying data to situate users. For example: how 
was it collected & what criteria we used to determine if research papers belonged to 
this corpus. If you have one or two more sentences you'd like to add to this effect, I'd 
be happy to update the description with those details. 

Thanks in advance for your review and look forward to hearing any feedback you may have. 


Dev 


On Fri, Mar 13, 2020 at 7:09 PM Parker, Lynne E. EOP/OSTP 


CC wrote: 


Hi Dev, 


Yes, I very much appreciate your tweaking the language to be more suitable. Please do that, 
and ГЇЇ review. 


Thanks again, 
Lynne 


On Mar 13, 2020, at 9:53 PM, Devvret Rishi BOE wrote: 


Hi Lynne, 


Thank you very much for the detailed note Lynne, and for sending the updated draft of 
questions. We have most of the updates to the dataset made (including mirroring the actual 
data on our platform) and research questions summarized as Tasks. You can see the 
continued progress on the landing page through this link. 


@Paul Mooney and I agreed with your assessment that certain questions were less of a fit 


for data mining approaches as currently written. We plan to adjust the language to some of 
them to make them better targeted problem statements for our community. For example, a 
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question on 'Develop animal models and standardize challenge studies' would 
become "Summarize efforts to develop animal models and standardize challenge 
studies" to be more tractable to the ML community based on published research. 
Please let us know if you have any concerns with the approach, and feel free to 
review any of them under the 'Tasks' tab. 


Thanks and looking forward to the launch! 
Dev 


On Fri, Mar 13, 2020 at 2:23 PM Parker, Lynne E. EOP/OSTP 
wrote: 


Dev, et al, 


Sure, sounds good. Please see attached for some of the scientific questions. These will not 
change, and are based on the WHO COVID 2019 R&D Blueprint. (They're almost the same as the 
WHO R&D Blueprint, but we did drop one or two that are clearly not text mining questions.) We 
are still working on getting another set from the National Academies. | was told | would have it 
about 3 hours ago, but it hasn't shown up yet. l'Il send that along as soon as possible. 


Note one of the sections of text l provided earlier: “Many of these questions are suitable for text 
mining, and we encourage researchers to develop text mining tools to provide insights on these 
questions.” Since these are drawn directly from the WHO R&D Blueprint (which isn't just about 
text mining existing literature), we recognize that some of these questions are likelv going to be 
difficult to answer bv text mining existing literature. We considered pruning down the list to 
those that seem more doable for text mining, but we don't want to preclude the possibilitv that 
someone can come up with a creative approach. (Or, mavbe awareness of the question is also 
informative.) So, we've included almost everything from the WHO R&D Blueprint. We value 
your judgement on this, though. If you think it unwise to include questions that seem difficult to 
address using text mining from existing literature, we're open to your thoughts. 


Thanks, 
Lynne 


From: Devvret Rishi 
Sent: Friday, March 13, 2020 5:03 PM 
To: Parker, Lynne E. EOP/OST 
Cc: Anthony Goldbloom 


; Sebastian 
; Julia Elliott 


Ben Hamner 
Subject: Re: [EXTERNAL] Re: COVID-19 AI Dataset 
Thanks Lynne, will make sure to add the updated reference to the roadmap as well. 

Through most of this afternoon, we'll be in meetings on how to best host this challenge, 


create tutorial content etc. I can update the landing page periodically, and have one up 
with the changes that have been suggested here over email this early evening PST if that 
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works for you? From there, I expect we'll want to continue to refine the copy and will be 
happy to help making changes there based on feedback from you and Carissa from the 
Allen Institute. 


Do you know when we should expect the finalized draft of questions? Once we have 
those, Paul will work on creating a Task on the dataset for each question or grouping of 
questions (some preliminary ones are up now as you may have seen). 


Best, 
Dev 


On Fri, Mar 13, 2020 at 1:42 PM Parker, Lynne E. EOP/OSTP 
wrote: 


Dev/Paul, 


We would like to add a link or reference to the WHO research roadmap, which is where some 
of the questions are coming from. See highlighted section below for this info. We will 
hopefully have a second link from the NASEM to provide soon. 


When do you think an updated landing page will be available for us to review? 


Lynne 


From: Parker, Lynne E. EOP/OSTP (б ШШШ 


Sent: Friday, March 13, 2020 1:01 PM 


To: 'Devvret Rishi' ФӘ >, Anthony Goldbloom 


Cc: Sebastian Kohlmeier БКБ) Н. Ben Hamner (BGE 
Parker, Lynne E. EOP/OSTP [Bey 


Subject: RE: [EXTERNAL] Re: COVID-19 AI Dataset 


Dev, 

Thanks again for the draft. A few notes: 

e  Let'sremove the "sponsored by" language in the header, since it seems to imply financial 
sponsorship. Would "hosted" make sense? 

e Please remove the White House from the header, and remove the WH logo. 


e The Chan Zuckerberg Institute should be added in the banner, and add their logo, and 
reference them in the acknowledgements. 


e COVID should be all caps. 
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epic.org 


e Don’t capitalize “pandemic”. 


e Inthe Introduction the language should read as follows: 


“We are issuing a call to action to the Nation’s artificial intelligence experts to develop new 
text and data mining tools that can help the medical community develop answers to high 
priority scientific questions. The CORE-19 dataset represents the most extensive machine- 
readable coronavirus literature collection available for data mining to date. This allows the 
worldwide Al research community the opportunity to apply text and data mining approaches 
to find answers to questions within, and connect insights across, this content in support of 
the ongoing COVID-19 response efforts worldwide.” 


e Add sentences in the acknowledgements that say: “Thanks also go to the World Health 

Organization and the National Academies of Sciences, Engineering, and Medicine’s Standing 

Committee on Emerging Infectious Diseases and 21st Century Health Threats for their expert 
work in defining the scientific questions outlined in this call to challenge. We also thank The 
White House Office of Science and Technology Policy for initiating and coordinating some of 

these efforts.” 


Also, on the questions/tasks, we would like to make the statement below somewhere to 
provide some context (maybe in an introduction to the tasks): 


“Informing this call to action, key scientific questions related to COVID-19 were obtained from 
the World Health Organization 


and the National Academies of Sciences, Engineering, and Medicine’s Standing 
Committee on Emerging Infectious Diseases and 21st Century Health Threats. Many of these 
questions are suitable for text mining, and we encourage researchers to develop text mining 
tools to provide insights on these questions.” 


As soon as we have final approval for the questions, I'll send them along. I’m hopeful it will 
be soon. 


That's it for now. We'll of course want to iterate on this, and may have more changes later 
(besides the questions). 


Lynne 


From: Devvret Rishi 
Sent: Friday, March 13, 2020 4:01 AM 
To: Anthony Goldbloom 


Cc: Sebastian Kohlmeier (EG, Parker, Lynne E. EOP/OSTP 
BST Kratsios, Michael J. EOP/OSTP 


8 
EPIC-20-03-24-OSTP-FOIA-202007 15-Production-pt3 000270 


; Ben Hamner (БІГУН 


Subject: Re: [EXTERNAL] Re: COVID-19 AI Dataset 


Thank you again very much for connecting today and providing a first draft of the 
research questions you are interested in. 


As Anthony mentioned, I wanted to send a prototype of what the data hosting & 
research questions may look like on Kaggle. Please note that all content is simply a 
placeholder right now, and will be updated once we have the finalized questions & 
dataset. We (and you) can also easily dynamically make updates as needed. 


1. You can access the dataset placeholder page using this link. 
2. You can then click the 'Tasks' button towards the top to see the list of open 
research questions for our ML community to answer. 
3. You can then click on individual tasks to read more, edit descriptions and see 
any submissions. 
I've also included a quick screen recording here that can show you the flow of seeing 
the dataset, the relevant questions (called Tasks on our platform) and placeholder 
submissions. 


As Anthony mentioned, once we are ready to release this to the public we can also do 
more work to promote this dataset & questions including a potential prize for strong 
solutions. Look forward to working with you on this and please let me know if you 
have any questions. 


Thanks, 
Dev 


On Thu, Mar 12, 2020 at 6:03 PM Anthony Goldbloom 
CC wrote: 


Lynne, thanks for sharing the questions. Sebastian, nice to connect and to work with 
you. 


Dev cc'd is the owner on our side. As a next step, he's going to share the prototype 
page (target tonight). 


Sebastian, agree it makes sense for us to cross reference each other's websites (our 
community might like using Semantic Scholar as a tool for exploring the data). As a 
next step, can you and Dev connect tomorrow to figure out the best way for Kaggle 
to get dumps of the raw data? We have an API, so ideal would be something 
programmatic. 


On Thu, Mar 12, 2020 at 4:08 PM Sebastian Kohlmeier KS- 


wrote: 
Hi Anthony, 
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epic.org 


Great to meet you and fantastic to see Kaggle involved in this important effort! 
From a logistics perspective we plan to publish the dataset along with a 
description and associated metadata (including licensing information) on this web 
page (currently just a placeholder until we release the new content). Since Kaggle 
will be hosting the questions it will probably be easiest for us to cross-link from 
Semantic Scholar to Kaggle and vice-versa for Monday's release. If that makes 
sense, it would be great if you could connect me with someone at Kaggle so that 
we can coordinate tomorrow and ensure that the right links and content are in 
place. 


Thanks again and excited to see this go live on Monday! 
Sebastian Kohlmeier 


Sr. Manager - Business Operations | Semantic Scholar 
Allen Institute for AI (А12) 


On Thu, Mar 12, 2020 at 3:46 PM Parker, Lvnne E. EOP/OSTP 


CC wrote 


Hi Anthony, et al, 


Please find attached a draft set of questions for the Kaggle site. Please treat this as a 
draft, for the purposes of your creating a prototype site, and don’t distribute until we 
give you the final version. We recognize that this is a long set of questions, but I’m of 
the opinion that in this current crisis, the more different lines of work that we offer to 
researchers, the more benefits we might gain. If this presents an unwieldy setup for 
Kaggle, however, then let me know and we can prune the list (but recognizing that 
prioritizing which questions are potentially most impactful is very difficult for us to 
do). 


I’m also copying Sebastian Kohlmeier from AllenAl, who can talk with you about the 
dataset. My understanding is that AllenAl would like to host the data, but would be 


fine with Kaggle mirroring it. 


We would like to go live with this effort on Monday, rather than tomorrow. When do 
you think you might have a draft site ready for us to review? 


Let me know if you need anything else just now. We very much appreciate your help 
and contribution to this effort! 


Lynne 
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RE: [EXTERNAL] Re: COVID-19 Open Research Dataset 
Press Call 


From "Hernandez, Elena В. EOP/OSTP" (Boy 
To: Devvret Rishi б ШШЕН. Anthony Goldbloom (БВ) ES 


Cc: "Kratsios, Michael J. EOP/OSTP" "Parker, Lynne E. 
і EOP/OSTP' 


Date: Sat, 14 Mar 2020 18:55:36 -0400 


Great, thank you so much! 


From: Devvret Rishi 
Sent: Saturday, March 14, 2020 6:50 PM 


To: Anthony Goldbloom (Dj 


Cc: Hernandez, Elena В. EOP/OSTP (БУВ) ИН ИН . Kratsios, Michael J. EOP/OSTP 


; Parker, Lynne E. EOP/OSTP (Bo) 


Subject: Re: [EXTERNAL] Re: COVID-19 Open Research Dataset Press Call 
Thank you Elena, Michael, and Lynne — really looking forward to this launch. 


The URL for the dataset and challenges which will be made public on Monday at 2pm EST 

://www. kaggle.com/allen-institute-for-ai/CORD-19-research-challenge«. We may make a 
short URL version of this before launch on Monday, in which case I will also circulate that with 
you asap. 


In case you're interested, you can view a staging (not for public release) version of the dataset 
here and the list of the tasks here. We're reviewing these iteratively with Lynne. 


Best, 
Dev 


On Sat, Mar 14, 2020 at 3:38 PM Anthony Goldbloom (GI wrote: 


Elena, adding dev from our team. He can share the URLs with you (to replace the placeholders 
in the current press release). 


On Sat, Mar 14, 2020, 11:04 AM Hernandez, Elena R. EOP/OSTP 


[E 


That's perfect, thanks Anthony for the quick turnaround! We are planning for 2pm on Monday for 
the call. 
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Will be back in touch soon with more details on the call and the latest version of the press release 
for your awareness, but let us know if you have any questions in the meantime. l can also be 


reached at (BN 


Elena 


Elena Hernandez 

Press Secretary 

Office of Science and Technology Policy 
The White House 


Эз | Mobil: (KG 
OO 


From: Anthony Goldbloom 
Sent: Saturday, March 14, 2020 1:56 PM 
To: Kratsios, Michael J. EOP/OSTP 


Cc: Parker, Lynne E. EOP/OSTP (б, Hernandez, Elena В. EOP/OSTP 


Subject: [EXTERNAL] Re: COVID-19 Open Research Dataset Press Call 


Michael for sure. 


Lynne and Elena, as far as quotes are concerned, does this work? I'd plan to say something 
similar on the teleconference. 


"It's difficult for humans to manually go through 24K- articles and synthesize 

their findings. Recent advances in the ability of machines to process natural language mean 
that AI might be helpful here. We're putting this dataset up in front of our community of 
4.3MM data scientists in the hope that the world's AI community can help find answers to a 
key set of questions about COVID-19." 


Anthony 


On Sat, Mar 14, 2020 at 8:36 AM Kratsios, Michael J. EOP/OSTP 


(ББ wrote 


Hi Anthony, 


We are so grateful for the work by your team to prepare the dataset and tasks for Monday. 
Having it up on your platform will make a really big difference. 


To issue our call to action and raise awareness across both the public and the scientific 
community, the we are hosting a media briefing call on Monday and putting out a press 
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release. I plan to give brief remarks and would love to have you dial-in and speak for a 
minute or two as well on kaggle's roll in this effort. Please let us know if you can join. 


Attached is a the draft press release and the comms plan. If you'd like to include a quote, 
feel free to send it to Lynne and Elena (cced) who are running point on the Monday roll- 
out. 


Thanks again for all your help! 


Best, 
Michael 


Michael Kratsios 

Deputy Assistant to the President 

Chief Technology Officer of the United States 
The White House 

0: 
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Re: [EXTERNAL] Re: COVID-19 Open Research Dataset Press 
Call 


Fom Devvret Rishi БУВ 
To: Anthony Goldbloom STI 


"Hernandez, Elena R. EOP/OSTP" "Kratsios, Michael 
Cc: J. EOP/OSTP" "Parker, Lynne E. EOP/OSTP" 


Date: Sat, 14 Mar 2020 18:50:00 -0400 
Thank you Elena, Michael, and Lvnne — really looking forward to this launch. 
The URL for the dataset and challenges which will be made public on Monday at 2pm EST 


is: »https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge«. We may make а 
short URL version of this before launch on Monday. in which case I will also circulate that with you asap. 


In case you're interested, you can view a staging (not for public release) version of the dataset here and the list of the 
tasks here. We're reviewing these iteratively with Lynne. 


Best, 
Dev 


On Sat, Mar 14, 2020 at 3:38 PM Anthony Goldbloom "(b(6  . | |. - wrote: 


Elena, adding dev from our team. He can share the URLs with you (to replace the placeholders in the current 
press release). 


On Sat, Mar 14, 2020, 11:04 AM Hernandez, Elena В. EOP/OSTP (B > wrote: 


That's perfect, thanks Anthony for the quick turnaround! We are planning for 2pm on Monday for 
the call. 


Will be back in touch soon with more details on the call and the latest version of the press release 
for your awareness, but let us know if you have any questions in the meantime. | can also be 


reached a BINTI 


Elena 


Elena Hernandez 

Press Secretary 

Office of Science and Technology Policy 
The White House 
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Desk ШОШ 
OO 


From: Anthony Goldbloom (BUTT 


Sent: Saturday, March 14, 2020 1:56 PM 


To: Kratsios, Michael J. ЕОР/ОЅТР О 
Cc: Parker, Lynne E. ЕОР/ОЅТР БОЈИ ИИ Hernandez, Elena В. EOP/OSTP 
> 


Subject: [EXTERNAL] Re: COVID-19 Open Research Dataset Press Call 
Michael for sure. 


Lynne and Elena, as far as quotes are concerned, does this work? I'd plan to say something similar on the 
teleconference. 


"It's difficult for humans to manually go through 24K- articles and synthesize their findings. Recent advances 
in the ability of machines to process natural language mean that AI might be helpful here. We're putting this 
dataset up in front of our community of 4.3MM data scientists in the hope that the world's AI community can 
help find answers to a key set of questions about COVID-19." 


Anthony 


On Sat, Mar 14, 2020 at 8:36 AM Kratsios, Michael J. EOP/OSTP PET wrote: 


Hi Anthony, 


We are so grateful for the work by your team to prepare the dataset and tasks for Monday. Having it up on 
your platform will make a really big difference. 


To issue our call to action and raise awareness across both the public and the scientific community, the we 
are hosting a media briefing call on Monday and putting out a press release. I plan to give brief remarks and 
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would love to have you dial-in and speak for a minute or two as well on kaggle's roll in this effort. Please let 
us know if you can join. 


Attached is a the draft press release and the comms plan. If you'd like to include a quote, feel free to send it 
to Lynne and Elena (cced) who are running point on the Monday roll-out. 


Thanks again for all your help! 


Best, 


Michael 


Michael Kratsios 

Deputy Assistant to the President 

Chief Technology Officer of the United States 
The White House 

0: 
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[EXTERNAL] Re: Monday COVID-19 Data/Al Announcement 


From «Fred Humphries (CELA) (БУВ 
To: "Kratsios, Michael J. EOP/OSTP" JG Karan Bhatia 


Cc: "Parker, Lynne E. EOP/OSTP' >, "Hernandez, Elena R. 
і EOP/OSTP' = 


Date: Sat, 14 Mar 2020 16:50:14 -0400 


Eric has kept me in the loop on this announcement, which is important as we deal with the public health issue. We 
are glad to participate. Thanks 


-Fred Humphries 


From: Kratsios, Michael J. EOP/OSTP (B6 


Sent: Saturday, March 14, 2020 3:31:41 PM 
To: Karan Bhatia Fred Humphries (CELA) 


Cc: Parker, Lynne E. EOP/OSTP (6 E Hernandez, Elena В. EOP/OSTP 


Subject: [EXTERNAL] Monday COVID-19 Data/Al Announcement 


Hi Karan and Fred, 


As was mentioned on the tech call last week, we have been working on releasing a COVID-19 related 
dataset. This is a first-of-its-kind, full-text, machine readable database of available scholarly literature 
about coronavirus. On Monday we'll be issuing a call to action to the Nation's artificial intelligence 
experts to develop new text and data mining tools that can help the scientific community develop 
answers to high-priority questions related to COVID-19. 


There will be a press call at 2pm on Monday. A draft press release is attached. 


Fred - | wanted to flag Microsoft Research has been involved and Eric Horvitz has agreed to be on the 
call and provided a quote. 

Karan — | wanted to flag that we have worked with Kaggle who will be hosting a copy of the dataset on 
their site so their community of data scientists can tackle some key questions re COVID-19. Anthony 
Goldbloom will join the call and provided a quote. 


Other Partners include: 
e Allen Institute for Al 
e Chan Zuckerberg Initiative 
e Georgetown University's Center for Security and Emerging Technology 
e NIH National Library of Medicine 


Many thanks for all your help! Just wanted to make sure you were in the loop with everything going on. 
Lynne and Elena (cced) can answer any questions you may have. 
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Best, 
Michael 


Michael Kratsios 

Deputy Assistant to the President 

Chief Technology Officer of the United States 
The White House 
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Monday COVID-19 Data/Al Announcement 
"Kratsios, Michael J. EOP/OSTP" e 


п» 


Егот: 


"s Karan Bhatia "Fred Humphries (CELA)" 


Cc: "Parker, Lynne E. EOP/OSTP" >, "Hernandez, Elena 
ġ R. EOP/OSTP' 
Date: Sat, 14 Mar 2020 15:31:41 -0400 


Attachment CORD-19 Dataset Press Release - OSTP А!2 CZI NIH KAGGLE MSFT.docx (21.16 
s: kB) 


Hi Karan and Fred, 


As was mentioned on the tech call last week, we have been working on releasing a COVID-19 related 
dataset. This is a first-of-its-kind, full-text, machine readable database of available scholarly literature 
about coronavirus. On Monday we'll be issuing a call to action to the Nation's artificial intelligence 
experts to develop new text and data mining tools that can help the scientific communitv develop 
answers to high-prioritv questions related to COVID-19. 


There will be a press call at 2pm on Monday. A draft press release is attached. 


Fred - | wanted to flag Microsoft Research has been involved and Eric Horvitz has agreed to be on the 
call and provided a quote. 

Karan — | wanted to flag that we have worked with Kaggle who will be hosting a copy of the dataset on 
their site so their community of data scientists can tackle some key questions re COVID-19. Anthony 
Goldbloom will join the call and provided a quote. 


Other Partners include: 
e Allen Institute for Al 
e Chan Zuckerberg Initiative 
e Georgetown University's Center for Security and Emerging Technology 
e  NIH National Library of Medicine 


Many thanks for all your help! Just wanted to make sure you were in the loop with everything going on. 
Lynne and Elena (cced) can answer any questions you may have. 


Best, 
Michael 


Michael Kratsios 


Deputy Assistant to the President 
Chief Technology Officer of the United States 
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The White House 


o(bX6) . 
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DRAFT 
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[EXTERNAL] Re: Thank You for Attending White House 


COVID-19 Tech Meeting 


From ^ Timothy Powderiy (BNI 
To: "Parker, Lynne E. ЕОР/О5ТР" (BN 


Cc: 'Kratsios, Michael J. EOP/OSTP' 'Hernandez, Elena R. 
` EOP/OSTP" 


Date: Sat, 14 Mar 2020 10:21:22 -0400 


Lvnne, 

Hope vou're well. We've received numerous requests from various entities (CDC, states) to provide support on a 
range of issues, including software development and hardware donations. At the moment we are working through 
those requests, as well as working on the items you outlined on our call this week. 


I will keep you updated on our progress. However, if there is a priority project the Administration is interested in, 
please let me know. 


Tim 


On Mar 11, 2020, at 4:51 PM, Parker, Lynne E. EOP/OSTP PET wrote: 


Good Afternoon, 


Thank you very much for being a part of today's COVID-19 Tech Meeting with The White House. We 
are appreciative for your expertise and collaboration. We are reaching out separately to entities who 
will be leading some of the next steps based on today's discussion. We will be connecting with you for 
continued engagement on these topics very soon. Please see the attached White House press release 
summarizing what was accomplished today and what is yet to come. 


We are grateful for your assistance in our nation's response to COVID-19. 
Lynne 
Lynne E. Parker, Ph.D. 


Deputy Chief Technology Officer of the United States 
The White House 


Office: ПВ 


«Mail Attachment.eml> 
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Dataset Comms for Tomorrow 


From: "Hernandez, Elena R. EOP/OSTP" Ее 


"Baum, Kristina В. EOP/OSTP" 
EOP/OSTP" 


"Bonyun, Sean C. 
P/OSTP" 


atson, lan D. 
>, "Waterman, Paige E. EOP/OSTP" 
>, "Lattimore, Tracie B. EOP/OSTP" 
, "Wallace, Rachel A. EOP/OSTP" 


Cc: "Parker, Lynne E. EOP/OSTP" ‚ "Kratsios, Michael J. 
Я EOP/OSTP' 


Date: Sun, 15 Mar 2020 22:26:53 -0400 


To: 


Attachment CORD19 Communications.docx (16.32 kB); media-briefing-invite.docx (12.88 kB); 
: CORD-19 Dataset Press Release - Final.docx (31.29 kB) 


s: 


Thank you 
Elena 
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THE WHITE HOUSE 
Office of Science and Technology Policy 
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THE WHITE HOUSE 
Office of Science and Technology Policy 
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DRAFT 


For more information about the novel coronavirus and COVID-19, please visit: 


https://www.cdc.gov/coronavirus 
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RE: [EXTERNAL] Re: covid 19 tech 


From 


"Parker, Lynne E. EOP/OSTP" (BU i 
To: Daniel Gross (B€: 


Сс: "Kratsios, Michael J. EOP/OSTP" >, "Parker, Lynne E. 
x EOP/OSTP" 


Date: Sun, 15 Mar 2020 12:41:40 -0400 


Hi Dan, 


Great talking with you. Below is the call-in info for the 3PM Eastern call today. l'Il also connect you 
separately to Josh. 


Lynne 


This event has been changed. 
COVID-19 Technology and Research Task Force 


When Sun Mar 15, 2020 12pm - 1pm Pacific Time - Los Angeles 
Calendar о 
Who (Guest list has been hidden at organizer's request) 


more details > 
Changed: Join Zoom Meeting 


Meeting ID: (ON 
Password: (b)(6) | 


One tap mobile 


GOMMA US (New York) 
GOMMA Us (San Jose) 


Dial bv vour location 


CLM (New York) 
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{БИБИ US (San Jose) 
Meeting ID: (ББ 
Find your local number: КВН 


Agenda 


3-3:05pm — National Science 8 Technologv Policv update 


Dr. Lvnne Partner, Deputv CTO, White House Office of Science and Technologv Policv 


3:05-3:20pm — CDC staff on prioritv needs 


Matthew Biggerstaff, Research Epidemiologist, Applied Research and Modeling Team, Influenza 
Division; Joe Bresee, Associate Director of Global Health Affairs, Influenza Division 


3:20-3:30pm — Geolocation data initiative update 


Dr. Caroline Buckee, Harvard School of Public Health 


3:30-3:35pm — Telehealth initiatives update 


Adam Greenberg, General Counsel, Ro 


3:35-3:55pm — Resourcing 


Group discussion with epidemiologv teams on what resources can be marshalled to fulfill needs 


3:55-4pm — Next steps 


Determine next steps for each initiative, and industrv contributions 
Going es ll Yes - Maybe - No more options > 


From: Daniel Gross 
Sent: Sunday, March 15, 2020 12:24 PM 


То: Kratsios, Michael J. EOP/OSTP (BUG) 
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Cc: Parker, Lynne E. EOP/OSTP (BIG) 


Subject: [EXTERNAL] Re: covid 19 tech 


Sure thing! Just dial me anytime. My number is SGI 


On Sun, Mar 15, 2020 at 6:14 AM Kratsios, Michael J. EOP/OSTP 


bom wot: 


Hi Daniel — I lead tech policy at the WH. I know you reached out to Jared yesterday on some 
ways the tech industry could be helpful. 


There is a group of private sector tech companies who have come together to see what data 
they have (e.g. geolocation, movement) which may be valuable in the modeling. They are 
doing a call today at 3pm — it may make sense or you to combine efforts. 


Do you have some time for a quick call with me or Lynne (our Dep US CTO) today? 


Best, 
Michael 


Michael Kratsios 

Chief Technology Officer of the United States 
The White House 

0: 

m: 
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RE: [EXTERNAL] Re: COVID-19 Open Research Dataset 
Press Call 


From «Hernandez, Elena В. ЕОР/ОЗТР' Б ИН 
To: Pete Voss (б EE li 


Anthony Goldbloom 


Pablo Chavez 
Devvret Rishi "Kratsios, Michael J. 
.gov>, "Parker, Lynne E. EOP/OSTP" 


Cc: 


Date: Sun, 15 Mar 2020 12:38:07 -0400 


That works for us — thanks! Will make sure that's reflected. 


From: Pete Vos 
Sent: Sunday, March 15, 2020 12:09 PM 
To: Hernandez, Elena R. EOP/OSTIB (GS 

Cc: Anthony Goldbloom Б Pablo Chavez (GING; Devvret 
Rishi SG; Kratsios, Michael J. EOP/OSTP <Michael.J.Kratsios@ostp.eop.gov>; 
Parker, Lynne E. EOP/OSTP 
Subject: Re: [EXTERNAL] Re: COVID-19 Open Research Dataset Press Call 


Thank you, Elena. I'm adding Pablo Chavez from our Google Cloud policy team for visibility. 


We would like to include the fact that Kaggle is owned by Google Cloud. Can we make the 
below edit to the Kaggle mention in the press release? 


"Through Kaggle, a machine learning and data science community owned by 
Google Cloud, these tools will be openly available for all researchers, including the 
medical research community." 


On Sun, Mar 15, 2020 at 8:25 AM Hernandez, Elena R. EOP/OSTP 


CC wrote: 


Good morning Kaggle team — here’s the latest version of the press release tomorrow. We are only 
waiting on a quote from Georgetown. Will be sure to let you know once we finalize. Thanks! 


From: Anthony Goldbloom 
Sent: Saturday, March 14, 2020 11:39 PM 
To: Dewret Rishi 
Cc: Hernandez, Elena R. EOP/OSTP Kratsios, Michael J. 


EOP/OSTP БКС; Parker, Lynne E. EOP/OSTP 
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Pete Voss (BGR > 
Subject: Re: [EXTERNAL] Re: COVID-19 Open Research Dataset Press Call 


Elena, some small changes to my quote. 


"It's difficult for people to manually go through more than 20,000 articles and synthesize their 
findings. Recent advances in technology can be helpful here. We're putting machine readable 
versions of these articles in front of our community of more than 4 million data scientists. Our 
hope is that AI can be used to help find answers to a key set of questions about COVID-19." 


Also adding Pete from Google's comms team. 


Anthony 


On Sat, Mar 14, 2020 at 3:50 PM Devvret Rishi Є wrote: 


Thank you Elena, Michael, and Lynne — really looking forward to this launch. 


The URL for the dataset and challenges which will be made public on Monday at 2pm EST 
iS: »»https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge««;. We may 
make a short URL version of this before launch on Monday, in which case I will also 
circulate that with you asap. 


In case you're interested, you can view a staging (not for public release) version of the 
dataset here and the list of the tasks here. We're reviewing these iteratively with Lynne. 


Best, 
Dev 


On Sat, Маг 14, 2020 at 3:38 PM Anthony Goldbloom (B)(6)9 wrote: 


Elena, adding dev from our team. He can share the URLs with you (to replace the 
placeholders in the current press release). 


On Sat, Mar 14, 2020, 11:04 AM Hernandez, Elena R. EOP/OSTP 


Co злого 


That’s perfect, thanks Anthony for the quick turnaround! We are planning for 2pm on 
Monday for the call. 


Will be back in touch soon with more details on the call and the latest version of the press 
release for your awareness, but let us know if you have any questions in the meantime. | can 


also be reached at (bX(6) ° 


Elena 


Elena Hernandez 
Press Secretary 
Office of Science and Technology Policy 
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epic.org 


The White House 


Desk: Mobile 


From: Anthony Goldbloom 
Sent: Saturday, March 14, 2020 1:56 PM 


To: Kratsios, Michael J. EOP/OSTP (Se 
Cc: Parker, Lynne E. EOP/OSTRBI UU Hernandez, Elena R. 


EOP/OSTP 
Subject: [EXTERNAL] Re: COVID-19 Open Research Dataset Press Call 


Michael for sure. 


Lynne and Elena, as far as quotes are concerned, does this work? I'd plan to say 
something similar on the teleconference. 


"It's difficult for humans to manually go through 24K+ articles and synthesize 

their findings. Recent advances in the ability of machines to process natural language 
mean that AI might be helpful here. We're putting this dataset up in front of our 
community of 4.3MM data scientists in the hope that the world's AI community can 
help find answers to a key set of questions about COVID- 19." 


Anthony 


On Sat, Mar 14, 2020 at 8:36 AM Kratsios, Michael J. EOP/OSTP 


оо 


Ні Anthony, 


We аге so grateful for (ће work Бу your team to prepare (ће dataset апа tasks for 
Monday. Having it up on your platform will make a really big difference. 


To issue our call to action and raise awareness across both the public and the 
scientific community, the we are hosting a media briefing call on Monday and 
putting out a press release. I plan to give brief remarks and would love to have you 
dial-in and speak for a minute or two as well on kaggle's roll in this effort. Please let 
us know if you can join. 


Attached is a the draft press release and the comms plan. If you'd like to include a 
quote, feel free to send it to Lynne and Elena (cced) who are running point on the 
Monday roll-out. 


Thanks again for all your help! 


3 
EPIC-20-03-24-OSTP-FOIA-202007 15-Production-pt3 000298 


Best, 
Michael 


Michael Kratsios 

Deputy Assistant to the President 

Chief Technology Officer of the United States 
The White House 

o: 


Pete Voss 


pM = AUML 
P6 — 
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[EXTERNAL] five reading links for March 


From: "Auren Hoffman (Five Links)" (б ШШШ 
To: 'Kratsios, Michael J. EOP/OSTP" (6e 


Date: Sun, 15 Mar 2020 12:15:32 -0400 


hello Michael... if you are stuck at home due to Coronavirus, here are five links 


worth reading ... 


Bonus: Coronavirus Tech Handbook & COVID-19 Readings 

These are open-sourced Google Docs for scholars, researchers, data scientists, 
and medical personnel. It links out to models, information, and more about the 
spread of COVID-19. In addition, this is a good COVID-19 dashboard. 


Metternich by Wolfram Siemann book review by Ferdinand Mount 
Who are the important 19th Century people to study? Certainly one can make 


the case that Metternich was in the top ten. This book review is so good, you 


might even be able to skip the book. 


Why “Exit Transparency” Can Make Companies Stronger by Auren Hoffman 
Your best-performing employee will eventually leave, either to start their own 
business or work for another company. One of the best things a one can do to 
manage exits is to have a blanket, well-defined “Exit Transparency” everyone in 


the company. 


The Gambler Who Cracked the Horse-Racing Code by Kit Chellel 
A wild tale of a gambler that might have made a $1 billion on algorithmically 


betting on the horses in Hong Kong. 
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